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Preface 


web2py was launched in 2007 and now, after four years of continuous 
development, we have reached a very much needed fourth edition of this 
book. During this time, web2py has managed to win the affection of thousands 
of knowledgeable users and more than one hundred developers. Our 
collective effort has created one of the most full-featured Open Source Web 
Frameworks in existence. 

I originally started web2py as a teaching tool because, I believe, the ability to 
build high quality web applications is of critical importance for the growth 
of a free and open society. It prevents the biggest players from monopolizing 
the flow of information. This motivation continues to be valid and it is even 
more important today. 

In general, the purpose of any web framework is to make web development 
easier, faster and prevent developers from making mistakes, especially in 
matters related to security. In web2py we address these issues with our three 
main goals: 

Ease of use is the primary goal for web2py. For us, this means reducing the 
learning and deployment time. This is why web2py is a full-stack framework 
without dependencies. It requires no installation and has no configuration 
files. Everything works out of the box, including a web server, database and a 
web-based IDE that gives access to all the main features. The API includes 
just 12 core objects, which are easy to work with and memorize. It can 
interoperate with most web servers, database and all Python libraries. 

Faster development is the secondary goal. Every function of web2py has a 
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default behavior (which can be overridden). For example, as soon as you have 
specified your data models, you will have access to a web-based database 
administration panel. web2py also generates automatically forms for your 
data and it allows you to easily expose the data in HTML, XML, JSON, RSS, 
etc. 

Security is at the heart of web2py, and our goal here is to lock everything 
down to keep your systems and data safe. Therefore, our database layer 
eliminates SQL Injections. The template language prevents Cross Site 
Scripting vulnerabilities. The forms generated by web2py provide field 
validation and block Cross Site Request Forgeries. Passwords are always 
stored hashed. Sessions are stored server-side by default to prevent Cookie 
Tampering and session cookies are uuid to prevent Cookie Stealing. 

web2py has always been built from the user perspective and is constantly 
optimized internally to become faster and leaner, whilst always maintaining 

backward compatibility. 

web2py is free for you to use. If you benefit from it, we hope you will feel a 
little more like contributing back to society in whatever form you choose. 

In 2011 InfoWorld magazine reviewed six of the most popular full-stack 
Python based web frameworks and raked web2py highest. Also in 2011, 
web2py won the Bossie Award for best Open Source Development Software. 


Introduction 


web2py [1] is a free, open-source web framework for agile development 
of secure database-driven web applications; it is written in Python [2] 
and programmable in Python. web2py is a full-stack framework, meaning 
that it contains all the components you need to build fully functional web 
applications. web2py is designed to guide a web developer to follow good 
software engineering practices, such as using the Model View Controller 
(MVC) pattern. web2py separates the data representation (the model) from 
the data presentation (the view) and also from the application logic and 
workflow (the controller). web2py provides libraries to help the developer 
design, implement, and test each of these three parts separately, and 
makes them work together. web2py is built for security. This means that 
it automatically addresses many of the issues that can lead to security 
vulnerabilities, by following well established practices. For example, it 
validates all input (to prevent injections), escapes all output (to prevent 
cross-site scripting), renames uploaded files (to prevent directory traversal 
attacks). web2py leaves little choice to application developers in matters 
related to security. web2py includes a Database Abstraction Layer (DAL) 
that writes SQL [3] dynamically so that you, the developer, don't have 
to. The DAL knows how to generate SQL transparently for SQLite [4], 
MySQL [6], PostgreSQL [5], MSSQL [7], FireBird [8], Oracle [9], IBM DB2 [10], 
Informix [11], and Ingres [12]. The DAL can also generate function calls for 
the Google Datastore when running on the Google App Engine (GAE) [13]. 
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Experimentally we support more databases. Please check on the web2py 
web site and mailing list for more recent adapters. Once one or more 
database tables are defined, web2py also generates a fully functional web- 
based database administration interface to access the database and the 
tables. web2py differs from other web frameworks in that it is the only 
framework to fully embrace the Web 2.0 paradigm, where the web is the 
computer. In fact, web2py does not require installation or configuration; 
it runs on any architecture that can run Python (Windows, Windows CE, 
Mac OS X, iOS, and Unix/Linux), and the development, deployment, and 
maintenance phases for the applications can be done via a local or remote 
web interface. web2py runs with CPython (the C implementation) and 
Jython (the Java implementation), on Python versions 2.4, 2.5, 2.6, and 2.7, 
although "officially" it only supports 2.5 so that we can guarantee backward 
compatibility for applications. web2py provides a ticketing system. If an 
error occurs, a ticket is issued to the user, and the error is logged for the 
administrator. web2py is open source and released under the LGPL version 
3 license. 

Another feature of web2py is that we, its developers, commit to maintain 
backward compatibility in future versions. We have done so since the first 
release of web2py in October, 2007. New features have been added and bugs 
have been fixed, but if a program worked with web2py 1.0, that program will 
still work today. 

Here are some examples of web2py statements that illustrate its power and 
simplicity. The following code: 

db.define_table( 'person' , Field (' name ') , Field ( 'image' , 'upload')) 

creates a database table called "person" with two fields: "name", a string; and 
"image", something that needs to be uploaded (the actual image). If the table 
already exists but does not match this definition, it is altered appropriately. 

Given the table defined above, the following code: 

form = crud.create(db. person) 

creates an insert form for this table that allows users to upload images. It 
also validates a submitted form, renames the uploaded image in a secure 
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way, stores the image in a file, inserts the corresponding record in the 
database, prevents double submission, and eventually modifies the form 
itself by adding error messages if the data submitted by the user does not 
pass validation. 

The following code: 

1 @auth. requires. permission ( ' read' , 'person' ) 
a def f(): .... 

prevents visitors from accessing the function f unless the visitor is a member 
of a group whose members have permissions to "read" records of table 
"person". If the visitor is not logged in, he gets directed to a login page 
(provided by default by web2py). 

The following code embeds a page component. 

1 {{=L0AD( ' ot he r-cont roller' , ' function, load' ,ajax=True, ajax_trap=True)}} 

This instructs web2py to load in a view the content generated by the other 
controller function (this works with any function). It loads the content via 
Ajax, embeds it into the current page (using the current layout, not the layout 
of the other_controller function), and traps all forms contained in the loaded 
content so that they are also submitted via Ajax without reloading the page. 
It can also LOAD content from non-web2py applications. 

The LOAD helper allows very modular design of applications; it is discussed 
in some detail in the last chapter of this book. 

1.1 Principles 

Python programming typically follows these basic principles: 

• Don't repeat yourself (DRY). 

• There should be only one way of doing things. 

• Explicit is better than implicit. 

web2py fully embraces the first two principles by forcing the developer to 
use sound software engineering practices that discourage repetition of code. 
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web2py guides the developer through almost all the tasks common in web 
application development (creating and processing forms, managing sessions, 
cookies, errors, etc.). 

web2py differs from other frameworks with regard to the third principle, 
which sometimes conflicts with the other two. In particular, web2py does 
not import user applications, but executes them in a predefined context. This 
context exposes the Python keywords, as well as the web2py keywords. 

To some this may appear as magic, but it should not. Simply, in practice, 
some modules are already imported without you doing so. web2py is 
trying to avoid the annoying characteristic of other frameworks that force 
the developer to import the same modules at the top of every model and 
controller. web2py, by importing its own modules, saves time and prevents 
mistakes, thus following the spirit of "don't repeat yourself" and "there 
should be only one way of doing things". 

If the developer wishes to use other Python modules or third-party modules, 
those modules must be imported explicitly, as in any other Python program. 


2 . 2 Web frameworks 

At its most fundamental level, a web application consists of a set of programs 
(or functions) that are executed when the corresponding URL is visited. The 
output of the program is returned to the visitor and rendered by the browser. 

The purpose of web frameworks is to allow developers to build new apps 
quickly, easily and without mistakes. This is done by providing APIs and 
tools that reduce and simplify the amount of coding that is required. 

The two classic approaches for developing web applications are: 

• Generating HTML [14, 15] programmatically. 

• Embedding code into HTML pages. 

The first model is the one that was followed, for example, by early CGI 
scripts. The second model is followed, for example, by PHP [16] (where the 
code is in PHP, a C-like language), ASP (where the code is in Visual Basic), 
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and JSP (where the code is in Java). 

Here is an example of a PHP program that, when executed, retrieves data 
from a database and returns an HTML page showing the selected records: 

<html><bodyxhl>Records</hl><? 

mysql_ connect (localhost, use rname, password) ; 

@mysql_select_db (database) or die( "Unable to select database") ; 

$query="S£LECr * FROM contacts"; 

$result=mysql_query($query) ; 

mysql_close( ) ; 

$i=0; 

while ($i < mysql_numrows($result) ) { 

$name=mysql_ result ($ res ult ,$i, "name") ; 

$phone=my sql_ result ($ res ult : $i, "phone" ) ; 

echo "<b>$name</bxbr>Phone:$phone<br /><br /><hr /><br />"; 

$i++; 
} 
?></bodyx/html> 

The problem with this approach is that code is embedded into HTML, but 
the very same code also needs to generate additional HTML and to generate 
SQL statements to query the database, entangling multiple layers of the 
application and making it difficult to read and maintain. The situation is 
even worse for Ajax applications, and the complexity grows with the number 
of pages (files) that make up the application. 

The functionality of the above example can be expressed in web2py with two 
lines of Python code: 

def index( ) : 

return HTML ( BODY ( HI ( 'Records' ) , db( ). select (db. contacts .ALL) ) ) 

In this simple example, the HTML page structure is represented 
programmatically by the HTML, BODY, and HI objects; the database db is queried 
by the select command; finally, everything is serialized into HTML. Notice 
that db is not a keyword but a user defined variable. We will use this name 
consistently to refer to a database connection to avoid confusion. 

Web frameworks are typically categorized as one of two types: A "glued" 
framework is built by assembling (gluing together) several third-party 
components. A "full-stack" framework is built by creating components 
designed specifically to be tightly integrated and work together. web2py is a 
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full-stack framework. Almost all of its components are built from scratch and 
are designed to work together, but they function just as well outside of the 
complete web2py framework. For example, the Database Abstraction Layer 
(DAL) or the template language can be used independently of the web2py 
framework by importing gluon.dal or gluon .template into your own Python 
applications, gluon is the name of the web2py module that contains system 
libraries. Some web2py libraries, such as building and processing forms from 
database tables, have dependencies on other portions of web2py. web2py 
can also work with third-party Python libraries, including other template 
languages and DALs, but they will not be as tightly integrated as the original 
components. 


2.3 Model-View-Controller 

web2py encourages the developer to separate data representation (the 
model), data presentation (the view) and the application workflow (the 
controller). Let's consider again the previous example and see how to build 
a web2py application around it. Here is an example of the web2py MVC edit 
interface: 


<- e H Q 127.0.0.1:80O0/admiri/defajlt/desit 

ir/sample 
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The typical workflow of a request in web2py is described in the following 
diagram: 


INTRODUCTION 27 


hup tf Vaclion | 


nup ui 1 jliC jl ■; inp'jl dipjV. I- 5: ji: 5 r: sijii SUIT I ■ Ji: sjujit.n 



DALinsen^updata^iejflf 
____DAL sele_ct 


dictionary 


HTMI. >K1I_ Pas JTV.M. J..~] .-!■- 


isrtrse map ul* ia^e Altaian cliae Iransaclian nan cron tasks 


^ HTTP Raspaf 


In the diagram: 

• The Server can be the web2py built-in web server or a third-party server, 
such as Apache. The Server handles multi-threading. 

• "main" is the main WSGI application. It performs all common tasks and 
wraps user applications. It deals with cookies, sessions, transactions, URL 
routing and reverse routing, and dispatching. 

It can serve and stream static files if the web server is not doing it already. 

• The Models, Views and Controller components make up the user 
application. 

• Multiple applications can be hosted in the same web2py instance. 

• The dashed arrows represent communication with the database engine(s). 
The database queries can be written in raw SQL (discouraged) or by using 
the web2py Database Abstraction Layer (recommended), so that web2py 
application code is not dependent on the specific database engine. 

• The dispatcher maps the requested URL to a function call in the controller. 
The output of the function can be a string or a dictionary of symbols (a 
hash table). The data in the dictionary is rendered by a view. If the visitor 
requests an HTML page (the default), the dictionary is rendered into an 
HTML page. If the visitor requests the same page in XML, web2py tries 
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to find a view that can render the dictionary in XML. The developer can 
create views to render pages in any of the already supported protocols 
(HTML, XML, JSON, RSS, CSV, RTF) or in additional custom protocols. 

• All calls are wrapped into a transaction, and any uncaught exception 
causes the transaction to be rolled back. If the request succeeds, the 
transaction is committed. 

• web2py also handles sessions and session cookies automatically, and when 
a transaction is committed, the session is also stored, unless specified 
otherwise. 

• It is possible to register recurrent tasks (via cron) to run at scheduled times 
and/or after the completion of certain actions. In this way it is possible to 
run long and compute-intensive tasks in the background without slowing 
down navigation. 

Here is a minimal and complete MVC application, consisting of three files: 

"db.py" is the model: 



It connects to the database (in this example a SQLite database stored in the 
storage. sqlite file) and defines a table called contact. If the table does not 
exist, web2py creates it and, transparently and in the background, generates 
SQL code in the appropriate SQL dialect for the specific database engine 
used. The developer can see the generated SQL but does not need to change 
the code if the database back-end, which defaults to SQLite, is replaced with 
MySQL, PostgreSQL, MSSQL, FireBird, Oracle, DB2, Informix, Interbase, 
Ingres, and the Google App Engine (both SQL and NoSQL). 

Once a table is defined and created, web2py also generates a fully functional 
web-based database administration interface, called appadmin, to access the 
database and the tables. 

' 'default. ,py" is the controller: 
def contacts ( ) : 
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z g rid=SQLFORM.g rid (db. contact, user_signature=False) 

3 return locals( ) 

In weD2py, URLs are mapped to Python modules and function calls. In this 
case, the controller contains a single function (or "action") called contacts. An 
action may return a string (the returned web page) or a Python dictionary 
(a set of key :value pairs) or the set of local variables (as in this example). If 
the function returns a dictionary, it is passed to a view with the same name 
as the controller /function, which in turn renders the page. In this example, 
the function contacts generates a select /search /create /update /delete grid 
for table db. contact and returns the grid to the view. 

"default/contacts. html" is the view. 

1 {{extend 'layout.html'}} 

2 <hl>Manage My Contacts</hl> 

3 {{=grid}} 

This view is called automatically by web2py after the associated controller 
function (action) is executed. The purpose of this view is to render the 
variables in the returned dictionary (in our case grid) into HTML. The view 
file is written in HTML, but it embeds Python code delimited by the special 
{{ and }} delimiters. This is quite different from the PHP code example, 
because the only code embedded into the HTML is "presentation layer" code. 
The "layout.html" file referenced at the top of the view is provided by web2py 
and constitutes the basic layout for all web2py applications. The layout file 
can easily be modified or replaced. 


1.4. Whyweb2py 

web2py is one of many web application frameworks, but it has compelling 
and unique features. web2py was originally developed as a teaching tool, 
with the following primary motivations: 

• Easy for users to learn server-side web development without 
compromising functionality. For this reason, web2py requires no 
installation and no configuration, has no dependencies (except for the 
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source code distribution, which requires Python 2.5 and its standard 
library modules), and exposes most of its functionality via a Web browser 
interface. 

web2py has been stable from day one because it follows a top-down 
design; i.e., its API was designed before it was implemented. Even as 
new functionality has been added, web2py has never broken backwards 
compatibility, and it will not break compatibility when additional 
functionality is added in the future. 

web2py proactively addresses the most important security issues which 
plague many modern web applications, as determined by OWASP [19] 
below. 

web2py is lightweight. Its core libraries, including the Database 
Abstraction Layer, the template language, and all the helpers amount to 
1.4MB. The entire source code including sample applications and images 
amounts to 10.4MB. 

web2py has a small footprint and is very fast. It uses the Rocket [22] 
WSGI web server developed By Timothy Farrell. It is 30% faster than 
Apache with mod_proxy. Our tests also indicate that, on an average PC, 
it serves an average dynamic page without database access in about 10ms. 
The DAL has very low overhead, typically less than 3%. 

web2py uses Python syntax for models, controllers, and views, but does 
not import models and controllers (as all the other Python frameworks 
do) - instead it executes them. This means that apps can be installed, 
uninstalled, and modified without having to restart the web server (even 
in production), and different apps can cohexist without their modules 
interfering with one another. 

web2py uses a Database Abstraction Layer (DAL) instead of an Object 
Relational Mapper (ORM). From a conceptual point of view, this means 
that different database tables are mapped into different instances of one 
Table class and not into different classes, while records are mapped into 
instances of one Row class, not into instances of the corresponding table 
class. From a practical point of view, it means that SQL syntax maps 
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almost one-to-one into DAL syntax, and there is no complex metaclass 
programming going on under the hood as in popular ORMs, which would 
add latency. 

WSGI [17, 18] (Web Server Gateway Interface) is an emerging Python 
standard for communication between a web server and Python applications). 

Here is a screenshot of the main web2py admin interface: 
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2.5 Security 

The Open Web Application Security Project [19] (OWASP) is a free and open 
worldwide community focused on improving the security of application 
software. 

OWASP has listed the top ten security issues that put web applications at 
risk. That list is reproduced here, along with a description of how each issue 
is addressed by web2py: 

• "Cross Site Scripting (XSS): XSS flaws occur whenever an application takes 
user supplied data and sends it to a web browser without first validating 
or encoding that content. XSS allows attackers to execute scripts in the 
victim's browser which can hijack user sessions, deface web sites, possibly 
introduce worms, etc." webipy, by default, escapes all variables rendered in the 
view, preventing XSS. 

• "Injection Flaws: Injection flaws, particularly SQL injection, are common 
in web applications. Injection occurs when user-supplied data is sent to 
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an interpreter as part of a command or query. The attacker's hostile data 
tricks the interpreter into executing unintended commands or changing 
data." webzpy includes a Database Abstraction Layer that makes SQL injection 
impossible. Normally, SQL statements are not written by the developer. Instead, 
SQL is generated dynamically by the DAL, ensuring that all inserted data is 
properly escaped. 

"Malicious File Execution: Code vulnerable to remote file inclusion 
(RFI) allows attackers to include hostile code and data, resulting in 
devastating attacks, such as total server compromise." webzpy allows only 
exposed functions to be executed, preventing malicious file execution. Imported 
functions are never exposed; only actions are exposed, webzpy uses a Web-based 
administration interface which makes it very easy to keep track of what is exposed 
and what is not. 

"Insecure Direct Object Reference: A direct object reference occurs when a 
developer exposes a reference to an internal implementation object, such 
as a file, directory, database record, or key, as a URL or form parameter. 
Attackers can manipulate those references to access other objects without 
authorization." webzpy does not expose any internal objects; moreover, webzpy 
validates all URLs, thus preventing directory traversal attacks, webzpy also 
provides a simple mechanism to create forms that automatically validate all input 
values. 

"Cross Site Request Forgery (CSRF): A CSRF attack forces a logged-on 
victim's browser to send a pre-authenticated request to a vulnerable web 
application, which then forces the victim's browser to perform a hostile 
action to the benefit of the attacker. CSRF can be as powerful as the web 
application that it attacks." webzpy prevents CSRF as well as accidental double 
submission of forms by assigning a one-time random token to each form. Moreover 
webzpy uses UUIDfor session cookie. 

"Information Leakage and Improper Error Handling: Applications 
can unintentionally leak information about their configuration, internal 
workings, or violate privacy through a variety of application problems. 
Attackers use this weakness to steal sensitive data, or conduct more 
serious attacks." webzpy includes a ticketing system. No error can result in 
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code being exposed to the users. All errors are logged and a ticket is issued to the 
user that allows error tracking. But errors and source code are accessible only to 
the administrator. 

"Broken Authentication and Session Management: Account credentials 
and session tokens are often not properly protected. Attackers 
compromise passwords, keys, or authentication tokens to assume other 
users' identities." webzpy provides a built-in mechanism for administrator 
authentication, and it manages sessions independently for each application. The 
administrative interface also forces the use of secure session cookies when the client 
is not "localhost" '. For applications, it includes a powerful Role Based Access 
Control API. 

"Insecure Cryptographic Storage: Web applications rarely use 
cryptographic functions properly to protect data and credentials. 
Attackers use weakly protected data to conduct identity theft and other 
crimes, such as credit card fraud." webzpy uses the MD5 or the HMAC+SHA- 
512 hash algorithms to protect stored passwords. Other algorithms are also 
available. 

"Insecure Communications: Applications frequently fail to encrypt 
network traffic when it is necessary to protect sensitive communications." 

webzpy includes the SSL-enabled [21] Rocket WSGI server, but it can also use 
Apache or Lighttpd and mod_ssl to provide SSL encryption of communications. 

"Failure to Restrict URL Access: Frequently an application only protects 
sensitive functionality by preventing the display of links or URLs to 
unauthorized users. Attackers can use this weakness to access and 
perform unauthorized operations by accessing those URLs directly." 

webzpy maps URL requests to Python modules and functions, webzpy provides 
a mechanism for declaring which functions are public and which require 
authentication and authorization. The included Role Based Access Control 
API allow developers to restrict access to any function based on login, group 
membership or group based permissions. The permissions are very granular and 
can be combined with CRUD to allow, for example, to give access to specific tables 
and/or records, webzpy also allows digitally signed URL and provides API to 
digitally sign ajax callbacks. 
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web2py was reviewed for security and you can find the result of the review 
in ref. [20]. 


1.6 In the box 

You can download web2py from the official web site: 

http://www.web2py.com 

web2py is composed of the following components: 

• libraries: provide core functionality of web2py and are accessible 
programmatically. 

• web server: the Rocket WSGI web server. 

• the admin application: used to create, design, and manage other 
web2py applications, admin provide a complete web-based Integrated 
Development Environment (IDE) for building web2py applications. It also 
includes other functionality, such as web-based testing and a web-based 
shell. 

• the examples application: contains documentation and interactive 
examples, examples is a clone of the official web2py.c0m web site, and 
includes epydoc documentation. 

• the welcome application: the basic scaffolding template for any other 
application. By default it includes a pure CSS cascading menu and user 
authentication (discussed in Chapter 9). 

web2py is distributed in source code, and in binary form for Microsoft 
Windows and for Mac OS X. 

The source code distribution can be used in any platform where Python 
runs and includes the above-mentioned components. To run the source 
code, you need Python 2.5 pre-installed on the system. You also need one 
of the supported database engines installed. For testing and light-demand 
applications, you can use the SQLite database, included with Python 2.5. 
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The binary versions of weD2py (for Windows and Mac OS X) include a 
Python 2.5 interpreter and the SQLite database. Technically, these two are not 
components of web2py Including them in the binary distributions enables 
you to run web2py out of the box. 

The following image depicts the overall web2py structure: 
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2.7 License 


web2py is licensed under the LGPL version 3 License. The full text of the 
license if available in ref. [31]. 

In accordance with LGPL you may: 

• redistribute web2py with your apps (including official web2py binary 
versions) 

• release your applications which use official web2py libraries under any 
license you wish 

Yet you must: 

• make clear in the documentation that your application uses web2py 

• release any modification of the web2py libraries under the LGPLV3 license 
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The license includes the usual disclaimer: 

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT 
PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE 
STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER 
PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF 
ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT 
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND 
FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE 
QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. 
SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST 
OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR 
AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR 
ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE 
PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR 
DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR 
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY 
TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS 
OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES 
SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE 
PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF 
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 
POSSIBILITY OF SUCH DAMAGES. 

Earlier versions 

Earlier versions of web2py, 1.0/-1.90.*, were released under the GPL2 license 
plus a commercial exception which, for practical purposes, was very similar 
to the current LPGLV3. 

Third party software distributed with web2py web2py contains third party 
software under the gluon/contrib/ folder and various JavaScript and CSS 
files. These files are distributed with web2py under their original licenses, as 
stated in the files. 
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1.9 About this book 


This book includes the following chapters, besides this introduction: 

• Chapter 2 is a minimalist introduction to Python. It assumes knowledge of 
both procedural and object-oriented programming concepts such as loops, 
conditions, function calls and classes, and covers basic Python syntax. It 
also covers examples of Python modules that are used throughout the 
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book. If you already know Python, you may skip Chapter 2. 

Chapter 3 shows how to start web2py, discusses the administrative 
interface, and guides the reader through various examples of increasing 
complexity: an application that returns a string, a counter application, an 
image blog, and a full blown wiki application that allows image uploads 
and comments, provides authentication, authorization, web services and 
an RSS feed. While reading this chapter, you may need to refer to Chapter 
2 for general Python syntax and to the following chapters for a more 
detailed reference about the functions that are used. 

Chapter 4 covers more systematically the core structure and libraries: URL 
mapping, request, response, sessions, caching, cron, internationalization 
and general workflow. 

Chapter 5 is a reference for the template language used to build views. It 
shows how to embed Python code into HTML, and demonstrates the use 
of helpers (objects that can generate HTML). 

Chapter 6 covers the Database Abstraction Layer, or DAL. The syntax of 
the DAL is presented through a series of examples. 

Chapter 7 covers forms, form validation and form processing. FORM is 
the low level helper for form building. SQLFORM is the high level form 
builder. In Chapter 7 we also discuss the Create/Read/Update/Delete 
(CRUD) API. 

Chapter 8 covers communication with as sending emails and SMSes. 

Chapter 9 covers authentication, authorization and the extensible Role- 
Based Access Control mechanism available in web2py. Mail configuration 
and CAPTCHA are also discussed here, since they are used for 
authentication. In the third edition of the book we have added extensive 
coverage of integration with third-party authentication mechanisms such 
as OpenID, OAuth, Google, Facebook, Linkedln, etc. 

Chapter 10 is about creating web services in web2py We provide examples 
of integration with the Google Web Toolkit via Pyjamas, and with Adobe 
Flash via PyAMF. 
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• Chapter 11 is about web2py and jQuery recipes. web2py is designed 
mainly for server-side programming, but it includes jQuery, since we have 
found it to be the best open-source JavaScript library available for effects 
and Ajax. In this chapter, we discuss how to effectively use jQuery with 
web2py. 

• Chapter 12 discusses web2py components and plugins as a way to build 
modular applications. We provide an example of a plugin that implements 
many commonly used functionality, such as charting, comments, tagging, 
and wiki. 

• Chapter 13 is about production deployment of web2py applications. We 
mainly address three possible production scenarios: on a Linux web 
server or a set of servers (which we consider the main deployment 
alternative), running as a service on a Microsoft Windows environment, 
and deployment on the Google Applications Engine. In this chapter, we 
also discuss security and scalability issues. 

• Chapter 14 contains a variety of other recipes to solve specific tasks, 
including upgrades, geocoding, pagination, the Twitter API, and more. 

This book only covers basic web2py functionalities and the API that ships 
with web2py This book does not cover web2py appliances (i.e. ready made 
applications). 

You can download web2py appliances from the corresponding web site [34] . 

You can find additional topics discussed on AlterEgo [35], the interactive 
web2py FAQ. 

This book has been written using the markmin syntax and automatically 
converted to HTML, LaTeX and PDF. 


2.20 Elements of style 

PEP8 [36] contains good style practices when programming with Python. 
You will find that web2py does not always follow these rules. This is not 
because of omissions or negligence; it is our belief that the users of web2py 
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should follow these rules and we encourage it. We chose not to follow some 
of those rules when defining web2py helper objects in order to minimize the 
probability of name conflict with objects defined by the user. 

For example, the class that represents a <div> is called DIV, while according 
to the Python style reference it should have been called Div. We believe that, 
for this specific example that using an all-upper-case "DIV" is a more natural 
choice. Moreover, this approach leaves programmers free to create a class 
called "Div" if they choose to do so. Our syntax also maps naturally into the 
DOM notation of most browsers (including, for example, Firefox). 

According to the Python style guide, all-upper-case strings should be 
used for constants and not variables. Continuing with our example, even 
considering that DIV is a class, it is a special class that should never 
be modified by the user because doing so would break other web2py 
applications. Hence, we believe this qualifies the DIV class as something that 
should be treated as a constant, further justifying our choice of notation. 

In summary, the following conventions are followed: 

• HTML helpers and validators are all upper case for the reasons discussed 
above (for example DIV, A, FORM, URL). 

• The translator object T is upper case despite the fact that it is an instance 
of a class and not a class itself. Logically the translator object performs 
an action similar to the HTML helpers, it affects rendering part of the 
presentation. Also, T needs to be easy to locate in the code and must have 
a short name. 

• DAL classes follow the Python style guide (first letter capitalized), for 
example Table, Field, Query, Row, Rows, etc. 

In all other cases we believe we have followed, as much as possible, the 
Python Style Guide (PEP8). For example all instance objects are lower-case 
(request, response, session, cache), and all internal classes are capitalized. 

In all the examples of this book, web2py keywords are shown in bold, while 
strings and comments are shown in italic. 


The Python language 


2.1 About Python 

Python is a general-purpose high-level programming language. Its design 
philosophy emphasizes programmer productivity and code readability. It 
has a minimalist core syntax with very few basic commands and simple 
semantics, but it also has a large and comprehensive standard library 
including an Application Programming Interface (API) to many of the 
underlying operating system (OS) functions. Python code, while minimalist, 
defines built-in objects such as linked lists (list), tuples (tuple), hash tables 
(diet), and arbitrarily long integers (long). 

Python supports multiple programming paradigms, including object- 
oriented (class), imperative (def), and functional (lambda) programming. 
Python has a dynamic type system and automatic memory management 
using reference counting (similar to Perl, Ruby, and Scheme). 

Python was first released by Guido van Rossum in 1991. The language has 
an open, community-based development model managed by the non-profit 
Python Software Foundation. There are many interpreters and compilers that 
implement the Python language, including one in Java (Jython) but, in this 
brief review, we refer to the reference C implementation created by Guido. 

You can find many tutorials, the official documentation and library references 
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of the language on the official Python website. [2] 

For additional Python references, we can recommend the books in ref. [37] 
and ref. [38]. 

You may skip this chapter if you are already familiar with the Python 
language. 


z.z Starting up 

The binary distributions of web2py for Microsoft Windows or Apple OS X 
come packaged with the Python interpreter built into the distribution file 
itself. 

You can start it on Windows with the following command (type at the DOS 

prompt): 

web2py.exe -S welcome 

On Apple OS X, enter the following command type in a Terminal window 
(assuming you're in the same folder as web2pyapp): 
. /web2py.app/Contents/MacOS/web2py -S welcome 

On a Linux or other Unix box, chances are that you have Python already 
installed. If so, at a shell prompt type: 
python web2py.py -S welcome 

If you do not have Python 2.5 (or later 2.x) already installed, you will have to 
download and install it before running web2py 

The -S welcome command line option instructs web2py to run the interactive 
shell as if the commands were executed in a controller for the welcome 
application, the web2py scaffolding application. This exposes almost all 
web2py classes, objects and functions to you. This is the only difference 
between the web2py interactive command line and the normal Python 
command line. 

The admin interface also provides a web-based shell for each application. 
You can access the one for the "welcome" application at. 
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http : //127 . . . 1 : 8000/admin/shell/index/welcome 

You can try all the examples in this chapter using the normal shell or the 
web-based shell. 


2.3 help, dir 

The Python language provides two commands to obtain documentation 
about objects defined in the current scope, both built-in and user-defined. 

We can ask for help about an object, for example "1": 

>» help(l) 

Help on int object: 

class int(object) 

int(x[, base]) -> integer 

Convert a string or number to an integer, if possible. A floating point 
argument will be truncated towards zero (this does not include a string 
representation of a floating point number!) When converting a string, use 
the optional base. It is an error to supply a base when converting a 
non-string. If the argument is outside the integer range a long object 
will be returned instead. 

Methods defined here: 

__abs__(...) 

x abs () <==> abs(x) 


and, since "1" is an integer, we get a description about the int class and all 
its methods. Here the output has been truncated because it is very long and 
detailed. 

Similarly, we can obtain a list of methods of the object "1" with the command 
dir: 


>» dir(l) 


['..abs^' , 

add ', ' and ', ' class ', ' cmp ', ' coerce ', 

•__delattr__ 

, ' div ', ' divmod ', ' doc ', ' float ', 

' floordiv— 

. ', ' getattribute ', ' getnewargs ', ' hash ', ' hex ', 

' index ' , 

' init ', ' int ', ' invert ', ' long ', ' Ishift ', 

' mod ' , '. 

__mul ', ' neg ', ' new ', ' nonzero ', ' oct ', 
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' or ' , '_ 

-00S_ 

- ' 1 

-DOW_ 

_', ' radd 

, 

rand— 

. ' , ' rdi V— 

. ' , 

' rdivmod 

, ' _ 

^ reduce ' , 

' reduce- 

ex ' , 

' repr ', ' rfloordiv ', 

'__rlshift__ 

1 , '_ 

_ rmod_ 

_ ' , 

rmul ' , 

' ror 

, ' 

rpow ' , '_. 

_rrshift__' , 

'__rs/iift— ' 

, ' 

rsub 

1 , '_ 

^rtruediv 

1 , ' 

rxor ' 

'setattr. 

' , 

'str__', ' 

sub 

— ' 1 

truediv ' , '_ 

_xor 

'] 




2.4 Types 

Python is a dynamically typed language, meaning that variables do not have 
a type and therefore do not have to be declared. Values, on the other hand, 
do have a type. You can query a variable for the type of value it contains: 



Python also includes, natively, data structures such as lists and dictionaries. 


2.4.I str 

Python supports the use of two different types of strings: ASCII strings 
and Unicode strings. ASCII strings are delimited by '...', "..." or by '..' or 
"""...""". Triple quotes delimit multiline strings. Unicode strings start with a u 
followed by the string containing Unicode characters. A Unicode string can 
be converted into an ASCII string by choosing an encoding for example: 


>» 

a 

= 

'this is 

an 

ASCII string 1 

>» 

b 

= 

u'This is 

a 

Unicode 

string' 

>» 

a 

= 

b.encode( 

' u 

tf8 l ) 



After executing these three commands, the resulting a is an ASCII string 
storing UTF8 encoded characters. By design, web2py uses UTF8 encoded 
strings internally. 
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It is also possible to write variables into strings in various ways: 



The last notation is more explicit and less error prone, and is to be preferred. 

Many Python objects, for example numbers, can be serialized into strings 
using str or repr. These two commands are very similar but produce slightly 
different output. For example: 

1 >» for i in [3, 'hello']: 

2 print str(i), repr(i) 

3 3 3 

4 hello 'hello' 

For user-defined classes, str and repr can be defined /redefined using the 

special operators str and repr__. These are briefly described later on; 

for more, refer to the official Python documentation [39]. repr always has a 
default value. 

Another important characteristic of a Python string is that, like a list, it is an 
iterable object. 

1 >» for i in 'hello' : 

2 print i 


3 h 

4 e 

5 I 

6 I 

7 


2.4.2 list 

The main methods of a Python list are append, insert, and delete: 
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Lists can be sliced: 



and concatenated: 



A list is iterable; you can loop over it: 

, >» a = [1, 2, 3] 

2 >» for i in a: 

3 print i 

4 1 

5 2 

6 3 

The elements of a list do not have to be of the same type; they can be any 
type of Python object. 

There is a very common situation for which a list comprehension can be used. 
Consider the following code: 
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This code clearly processes a list of items, selects and modifies a subset of the 
input list, and creates a new result list, and this code can be entirely replaced 
with the following list comprehension: 


>» a = 

[1,2,3,4,5] 


>» b = 

[x * 3 for x in a if x % 2 

== 0] 

>» b 



[6, 12] 




2.4.3 tuple 

A tuple is like a list, but its size and elements are immutable, while in a list 
they are mutable. If a tuple element is an object, the object attributes are 
mutable. A tuple is delimited by round brackets. 

>» a = (1, 2, 3) 

So while this works for a list: 

>» a = [1, 2, 3] 
>» a[l] = 5 
>» print a 
[1, 5, 3] 

the element assignment does not work for a tuple: 



A tuple, like a list, is an iterable object. Notice that a tuple consisting of a 
single element must include a trailing comma, as shown below: 
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Tuples are very useful for efficient packing of objects because of their 
immutability, and the brackets are often optional: 

>» a = 2, 3, 'hello' 

>» x, y, z = a 

>» print x 

2 

>» print z 

hello 


2.4.4 dlct 

A Python dict-ionary is a hash table that maps a key object to a value object. 
For example: 

, >» a = {' k' : ' v' , ' k2' :3} 

2 »> a [ ' k ' ] 

3 v 

4 >» a [ ' k2 ' ] 

5 3 

6 >» a.has_key( ' k' ) 

7 True 

8 >» a.has_key( ' v' ) 
,., False 

Keys can be of any hashable type (int, string, or any object whose class 

implements the hash method). Values can be of any type. Different keys 

and values in the same dictionary do not have to be of the same type. If the 
keys are alphanumeric characters, a dictionary can also be declared with the 
alternative syntax: 



Useful methods are has_key, keys, values and items: 
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6 >» print a. items () 

7 ICk 1 , '!/'), Vk2\ 3)] 


The items method produces a list of tuples, each containing a key and its 
associated value. 

Dictionary elements and list elements can be deleted with the command del: 



Internally Python uses the hash operator to convert objects into integers, and 
uses that integer to determine where to store the value. 


>» hash("heZZo world") 
-1500746465 


2.5 About indentation 


Python uses indentation to delimit blocks of code. A block starts with a 
line ending in colon, and continues for all lines that have a similar or higher 
indentation as the next line. For example: 


>» 

i 

= 

>» 

wh 

ile i < 3: 

>» 


print i 

>» 


i = i + 1 

>» 







1 



2 




It is common to use four spaces for each level of indentation. It is a good 
policy not to mix tabs with spaces, which can result in (invisible) confusion. 
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2.6 for... in 



In Python, you 

can 

loop over 

iterable 

objects: 

1 

>» a = [0, 1, '1 

lello 

1 , 'python' 

] 


2 

>» for i in a: 





3 
4 
5 

print i 

1 





6 
7 

hello 
python 






One common shortcut is xrange, which generates an iterable range without 
storing the entire list of elements. 

1 >» for i in xrange(0, 4): 

2 print i 

3 

4 1 

5 2 

6 3 

This is equivalent to the C/C++/C#/Java syntax: 
i for(int i=0; i<4; i=i+l) { print(i); } 



Another useful command is enumerate, which counts while looping: 

1 

>» a = [0, 1, 'hello', 'python'] 

2 

>» for i, j in enumerate(a) : 

3 

print i, j 

4 



5 

1 1 

6 

2 hello 

7 

3 python 


There is also a keyword range (a, b, c) that returns a list of integers starting 
with the value a, incrementing by c, and ending with the last value smaller 
than b, a defaults to o and c defaults to l. xrange is similar but does not 
actually generate the list, only an iterator over the list; thus it is better for 
looping. 

You can jump out of a loop using break 

>» for i in [1, 2, 3] : 

print i 
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break 

1 

You can jump to the next loop iteration without executing the entire code 
block with continue 

>» for i in [1, 2, 3] : 

print i 
continue 
print 'test' 
1 


5 
2 

3 


2. J while 

The while loop in Python works much as it does in many other programming 
languages, by looping an indefinite number of times and testing a condition 
before each iteration. If the condition is False, the loop ends. 



There is no loop . . . until construct in Python. 
2.8 if . . .elif . . .else 



The use of conditionals in Python is intuitive: 

1 

>» for 

i in range(3) : 

2 

»> 

if i == 0: 

3 

»> 

print 'zero' 

4 

»> 

elif i == 1: 

5 

»> 

print 'one' 

6 

»> 

else: 

7 

»> 

print 'other' 

8 

zero 


9 

one 
other 
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"elif" means "else if". Both elif and else clauses are optional. There can be 
more than one elif but only one else statement. Complex conditions can be 
created using the not, and and or operators. 


>» for i in range(3) : 


>» if i == or (i == 

1 and i + 1 == 2) : 

>» print '0 or 1' 



2.C) try. . .except. . .else. . .finally 

Python can throw - pardon, raise - Exceptions: 



If the exception is raised, it is caught by the except clause, which is executed, 
while the else clause is not. If no exception is raised, the except clause is not 
executed, but the else one is. The finally clause is always executed. 

There can be multiple except clauses for different possible exceptions: 

i >» try: 

2 >» raise SyntaxError 

3 >» except ValueError: 

4 >» print 'value error' 

5 >» except SyntaxError: 

6 >» print 'syntax error' 

7 syntax error 


The else and finally clauses are optional. 

Here is a list of built-in Python exceptions + HTTP (defined by web2py) 


BaseException 
+-- HTTP (defined by web2py) 
+-- SystemExit 
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Keyboard Interrupt 
Exception 


6 

4--- GeneratorExit 

7 

4--- Sto 

plteration 

8 

4--- StandardError 

9 

1 +- 
1 1 
1 1 
1 1 

- ArithmeticError 

+-- FloatingPointError 
+-- OverflowError 
+-- ZeroDivisionError 

13 

1 + " 

- AssertionError 

H 

I + " 

- AttributeError 

15 

1 + " 

- EnvironmentError 

16 

1 1 

+-- IOError 

*7 

1 1 

+-- OSError 

18 

1 1 

4--- WindowsError (Windows) 

*9 

1 1 

4--- VMSError (VMS) 

20 

1 + " 

- EOFError 

21 

i + " 

- ImportError 

22 

1 + " 

- LookupError 

23 

1 1 

+-- IndexError 

24 

1 1 

+-- KeyError 

25 

1 + " 

- MemoryError 

26 

1 + " 

- NameError 

27 

1 1 

+-- UnboundLocalError 

28 

1 + " 

- ReferenceError 

29 

1 + " 

- RuntimeError 

30 

1 1 

+-- WotlmplementedError 

31 

1 +_ 

- SyntaxError 

32 

1 1 

+-- IndentationError 

33 

1 1 

4--- TabError 

34 

| +- 

- SystemError 

35 

1 + " 

- TypeError 

36 

1 + " 

- ValueError 

37 

1 1 

+-- UnicodeError 

38 

1 1 

4--- UnicodeDecodeError 

39 

1 1 

4--- UnicodeEncodeError 

40 

1 1 

4--- UnicodeTranslateError 

4 1 

4--- Warning 

42 

+ - 

- DeprecationWarning 

43 

4-- 

- PendingDeprecationWarning 

44 

+ - 

- RuntimeWarning 

45 

+- 

- SyntaxWarning 

46 

+ - 

- UserWarning 

47 

+ - 

- FutureWarning 

48 

4-- - Im 

portWarning 

49 

4--- Un 

icodeWarning 
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For a detailed description of each of them, refer to the official Python 
documentation. web2py exposes only one new exception, called HTTP. When 
raised, it causes the program to return an HTTP error page (for more on this 
refer to Chapter 4). 

Any object can be raised as an exception, but it is good practice to raise 
objects that extend one of the built-in exception classes. 


2.20 def . . . return 

Functions are declared using def. Here is a typical Python function: 

>» def f (a, b) : 

return a + b 
>» print f(4 : 2) 
6 

There is no need (or way) to specify types of the arguments or the return 
type(s). In this example, a function f is defined that can take two arguments. 

Functions are the first code syntax feature described in this chapter to 
introduce the concept of scope, or namespace. In the above example, the 
identifiers a and b are undefined outside of the scope of function f : 



Identifiers defined outside of function scope are accessible within the 
function; observe how the identifier a is handled in the following code: 
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If a is modified, subsequent function calls will use the new value of the global 
a because the function definition binds the storage location of the identifier 
a, not the value of a itself at the time of function declaration; however, if a 
is assigned-to inside function g, the global a is unaffected because the new 
local a hides the global value. The external-scope reference can be used in 
the creation of closures: 


1 

>» 

def f (x) : 

2 


def g(y): 

3 


return x * y 

4 


return g 

5 

»> 

doubter = f(2) # doubler is a new function 

6 

»> 

tripter = f(3) # tripler is a new function 

7 

»> 

quadrupter = f(4) # quadrupler is a new function 

8 

>» 

print doubter(5) 

9 

10 


[0 

>» 

print tripler(5) 

[i 

15 


12 

>» 

print quadrupler(5) 

13 

20 



Function f creates new functions; and note that the scope of the name g is 
entirely internal to f . Closures are extremely powerful. 

Function arguments can have default values, and can return multiple results: 

, >» def f (a, b=2): 

z return a + b, a - b 

i >» x, y = f(5) 

4 >» print x 

5 7 

6 >» print y 

7 3 
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Function arguments can be passed explicitly by name, and this means that 

the order of arguments specified in the caller can be different than the order 

of arguments with which the function was defined: 

1 >» def f (a, b=2): 

z return a + b, a - b 

3 >» x, y = f(b=5, a=2) 

4 >» print x 

5 7 

6 >» print y 

7 -3 



Functions can also take a runtime-variable number of arguments: 

1 

>» def f (*a, **b) : 

2 

return a, b 

3 

>» x, y = f(3 : 'hello', c=4 : test=' world' ) 

4 

>» print x 

5 

(3, 'hello') 

6 

»> print y 

7 

{ ' c' -A, ' test' : 'world' } 


Here arguments not passed by name (3, 'hello') are stored in the tuple a, and 
arguments passed by name (c and test) are stored in the dictionary b. 

In the opposite case, a list or tuple can be passed to a function that requires 
individual positional arguments by unpacking them: 

>» def f (a, b) : 

return a + b 
>» c = (1, 2) 
>» print f(*c) 
3 

and a dictionary can be unpacked to deliver keyword arguments: 


>» 

def 

f(a. 

b): 





return a 

+ 

b 

>» 

c = 

{'a' 

:1, 

'£> 

:2} 

>» 

print f( 

**c) 



3 







2.10.1 lambda 

lambda provides a way to create a very short unnamed function very easily: 
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>» a = lambda b: b + 2 

>» print a(3) 

5 

The expression "lambda [a]:[b]" literally reads as "a function with arguments 
[a] that returns [b]". The lambda expression is itself unnamed, but the function 
acquires a name by being assigned to identifier a. The scoping rules for def 
apply to lambda equally, and in fact the code above, with respect to a, is 
identical to the function declaration using def: 


>» def a(b) : 

return b + 2 
>» print a(3) 

5 


The only benefit of lambda is brevity; however, brevity can be very convenient 
in certain situations. Consider a function called map that applies a function to 
all items in a list, creating a new list: 


>» 

a 

= [1, 7, 

2, 5, 

4, 

8] 

>» 

me 

ip(lambda 

x: x 

+ 2 

, a) 

[3, 

9, 

4, 7, 6 

, 10] 




This code would have doubled in size had def been used instead of lambda. 
The main drawback of lambda is that (in the Python implementation) the 
syntax allows only for a single expression; however, for longer functions, 
def can be used and the extra cost of providing a function name decreases as 
the length of the function grows. Just like def, lambda can be used to curry 
functions: new functions can be created by wrapping existing functions such 
that the new function carries a different set of arguments: 

>» def f(a, b) : return a + b 
>» g = lambda a: f(a, 3) 
>» g(2) 

5 

There are many situations where currying is useful, but one of those is 
directly useful in web2py: caching. Suppose you have an expensive function 
that checks whether its argument is prime: 


def isprime(number) : 

for p in range(2, number): 
if (number % p) == 0: 
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5 


return False 
return True 


This function is obviously time consuming. 

Suppose you have a caching function cache, ram that takes three arguments: a 
key, a function and a number of seconds. 
1 value = cache. ram ( 'key ' , f, 60) 

The first time it is called, it calls the function f ( ) , stores the output in a 
dictionary in memory (let's say "d"), and returns it so that value is: 

i value = d[ 'key ' ]=f ( ) 

The second time it is called, if the key is in the dictionary and not older 
than the number of seconds specified (60), it returns the corresponding value 
without performing the function call. 

1 value = d[ 'key ' ] 

How would you cache the output of the function isprime for any input? Here 
is how: 


>» number = 7 




>» seconds = 60 




>» print cache. ram(str(number) , 

. lambda: 

isprime(number) , 

. seconds) 

True 




>» print cache. ram(str(number) , 

, lambda: 

isprime(number) , 

, seconds) 

True 





The output is always the same, but the first time cache, ram is called, isprime 
is called; the second time it is not. 

Python functions, created with either def or lambda allow re-factoring existing 
functions in terms of a different set of arguments, cache, ram and cache. disk 
are webzpy caching functions. 


2.11 class 

Because Python is dynamically typed, Python classes and objects may seem 
odd. In fact, you do not need to define the member variables (attributes) 
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when declaring a class, and different instances of the same class can have 
different attributes. Attributes are generally associated with the instance, not 
the class (except when declared as "class attributes", which is the same as 
"static member variables" in C++ /Java). 

Here is an example: 

i >» class MyClass(object) : pass 

2 >» myinstance = MyClassO 

3 >» myinstance. myvariable = 3 

4 >» print myinstance. myvariable 

5 3 

Notice that pass is a do-nothing command. In this case it is used to define a 
class MyClass that contains nothing. MyClass ( ) calls the constructor of the class 
(in this case the default constructor) and returns an object, an instance of the 
class. The (object) in the class definition indicates that our class extends the 
built-in object class. This is not required, but it is good practice. 


Here is a more complex class: 

i >» 

class MyClass(object) : 

2 >» 

z = 2 

3 >» 

def init (self, a, b) : 

4 >» 

self .x = a, self .y = b 

>» 

def add(self): 

6 >» 

return self.x + self.y + self.z 

7 »> 

myinstance = MyClass(3, 4) 

S >» 

print myinstance. add( ) 

9 9 



Functions declared inside the class are methods. Some methods have special 

reserved names. For example, init is the constructor. All variables are 

local variables of the method except variables declared outside methods. For 
example, z is a class variable, equivalent to a C++ static member variable that 
holds the same value for all instances of the class. 

Notice that init takes 3 arguments and add takes one, and yet we call 

them with 2 and o arguments respectively. The first argument represents, 
by convention, the local name used inside the method to refer to the current 
object. Here we use self to refer to the current object, but we could have used 
any other name, self plays the same role as *this in C++ or this in Java, but 
self is not a reserved keyword. 
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This syntax is necessary to avoid ambiguity when declaring nested classes, 
such as a class that is local to a method inside another class. 


2.22 Special attributes, methods and operators 

Class attributes, methods, and operators starting with a double underscore 
are usually intended to be private (i.e. to be used internally but not exposed 
outside the class) although this is a convention that is not enforced by the 
interpreter. 

Some of them are reserved keywords and have a special meaning. 

Here, as an example, are three of them: 

• __len__ 

• getitem 

• setitem 

They can be used, for example, to create a container object that acts like a list: 


>» 

class MyList (object) : 




>» 

def init (self 

, *a 

): 

self. a = list(a) 

>» 

def __len__(self) 

: return len(self.a) 

>» 

def getitem (s 

elf, 

i) 

: return self.a[i] 

>» 

def setitem (s 

elf, 

i, 

i): self.a[i] = j 

>» 

b = MyList(3, 4, 5) 




>» 

4 

>» 

print b[l] 




b.a[l] = 7 




>» 

print b.a 




[3, 

7, 5] 





Other special operators include getatt r and setatt r , which define the 

get and set attributes for the class, and sum and sub , which overload 

arithmetic operators. For the use of these operators we refer the reader to 
more advanced books on this topic. We have already mentioned the special 
operators str and repr__. 
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2.23 File input/output 

In Python you can open and write in a file with: 

1 >» file = open ( 'myfile. txt' , V) 

2 >» file. write( 'hello world') 

3 >» file.close() 

Similarly you can read back from the file with: 

1 >» file = open ( 'myfile. txt ' , ' /-') 

2 >» print file, read () 
i hello world 

Alternatively you can read in binary mode with "rb", write in binary mode 
with "wb", and open the file in append mode "a", using standard C notation. 

The read command takes an optional argument, which is the number of bytes. 
You can also jump to any location in a file using seek. 

You can read back from the file with read 

1 >» print file.seek(6) 

2 >» print file, read () 
, world 

and you can close the file with: 
] >» file.close() 

In the standard distribution of Python, which is known as CPython, variables 
are reference-counted, including those holding file handles, so CPython knows 
that when the reference count of an open file handle decreases to zero, the file 
may be closed and the variable disposed. However, in other implementations of 
Python such as PyPy, garbage collection is used instead of reference counting, 
and this means that it is possible that there may accumulate too many open file 
handles at one time, resulting in an error before the gc has a chance to close 
and dispose of them all. Therefore it is best to explicitly close file handles when 
they are no longer needed, webzpy provides two helper functions , read^f ile( ) 
and write_file() inside the gluon.fileutils namespace that encapsulate the 
file access and ensure that the file handles being used are properly closed. 
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When using webipy, you do not know where the current directory is, because 
it depends on how webzpy is configured. The variable request . folder contains 
the path to the current application. Paths can be concatenated with the 
command os . path . j oin, discussed below. 


2.1Q. exec, eval 

Unlike Java, Python is a truly interpreted language. This means it has the 
ability to execute Python statements stored in strings. For example: 

, >» a = "print 'hello world'" 
2 >» exec (a) 
-, 'hello world' 


What just happened? The function exec tells the interpreter to call itself and 
execute the content of the string passed as argument. It is also possible to 
execute the content of a string within a context defined by the symbols in a 
dictionary: 


>» a = "print b" 
>» c = dict(b=3) 
>» exec (a, {}, c) 
3 


Here the interpreter, when executing the string a, sees the symbols defined 
in c (b in the example), but does not see c or a themselves. This is different 
than a restricted environment, since exec does not limit what the inner code 
can do; it just defines the set of variables visible to the code. 

A related function is eval, which works very much like exec except that it 
expects the argument to evaluate to a value, and it returns that value. 

1 >» a = "3*4" 

a >» b = eval(a) 

3 >» print b 

4 12 
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2.25 import 

The real power of Python is in its library modules. They provide a large and 
consistent set of Application Programming Interfaces (APIs) to many system 
libraries (often in a way independent of the operating system). 

For example, if you need to use a random number generator, you can do: 

1 >» import random 

2 »> print random. randint(0, 9) 

3 5 

This prints a random integer between o and 9 (including 9), 5 in the example. 
The function randint is defined in the module random. It is also possible to 
import an object from a module into the current namespace: 

1 >» from random import randint 

2 >» print randint(0, 9) 


or 


import all objects from a module into the current namespace: 


, >» from random import * 
2 >» print randint(0, 9) 


or import everything in a newly defined namespace: 

>» import random as myrand 
>» print myrand. randint(0, 9) 

In the rest of this book, we will mainly use objects defined in modules os, 
sys, datetime, time and cPickle. 

All of the webzpy objects are accessible via a module called gluon, and that is 
the subject of later chapters. Internally, webzpy uses many Python modules 
(for example thread), but you rarely need to access them directly. 

In the following subsections we consider those modules that are most useful. 


2.15.1 os 

This module provides an interface to the operating system API. For example: 
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>» import os 

>» os.chdir( ' . . ' ) 

>» os. unlink ( ' filename_to_be_deleted' ) 

Some of the os functions, such as chdir, MUST NOT be used in webzpy 
because they are not thread-safe. 

os. path. join is very useful; it allows the concatenation of paths in an OS- 
independent way: 

>» import os 

>» a = os . path .join ( 'path' , ' sub_path' ) 

>» print a 

path/sub_path 

System environment variables can be accessed via: 
>» print os. environ 


which is a read-only dictionary. 


2.15.2 sys 

The sys module contains many variables and functions, but the one we use 
the most is sys. path. It contains a list of paths where Python searches for 
modules. When we try to import a module, Python looks for it in all the 
folders listed in sys. path. If you install additional modules in some location 
and want Python to find them, you need to append the path to that location 
to sys .path. 

1 >» import sys 

2 >» sys. path. append ( 'path/to/my/modules' ) 

When running web2py, Python stays resident in memory, and there is only 
one sys. path, while there are many threads servicing the HTTP requests. To 
avoid a memory leak, it is best to check if a path is already present before 
appending: 

1 >» path = 'path/to/my/modules' 

2 >» if not path in sys. path: 

3 sys. path. append(path) 
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2.15.3 datetime 

The use of the datetime module is best illustrated by some examples: 


>» 

import 

datetime 




>» 

print 

datetime. datet 

:ime 

.today 



2008 

-07-04 

14:03:90 




>» 

print 

datetime. date. 

today! ) 


2008 

;-07-04 






Occasionally you may need to time-stamp data based on the UTC time as 
opposed to local time. In this case you can use the following function: 


1 >» import datetime 

z >» print datetime. datetime. utcnowf ) 

3 2008-07-04 14:03:90 


The datetime module contains various classes: date, datetime, time and 
timedelta. The difference between two date or two datetime or two time 
objects is a timedelta: 



In web2py, date and datetime are used to store the corresponding SQL types 
when passed to or returned from the database. 


2.15.4 time 

The time module differs from date and datetime because it represents time as 
seconds from the epoch (beginning of 1970). 

>» import time 
>» t = time. time( ) 
1215138737.571 

Refer to the Python documentation for conversion functions between time in 
seconds and time as a datetime. 
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2.15.5 cPickle 

This is a very powerful module. It provides functions that can serialize 
almost any Python object, including self-referential objects. For example, 
let's build a weird object: 


>» class MyClass(object) : pass 




>» myinstance = MyClassO 




>» myinstance. x = 'something' 




>» a = [1 ,2, {' hello' :' world'}, 

[3, 

4, 

[myinstance] ] ] 


and now: 

>» import cPickle 

>» b = cPickle. dumps(a) 

>» c = cPickle. loads(b) 


In this example, b is a string representation of a, and c is a copy of a generated 
by de-serializing b. cPickle can also serialize to and de-serialize from a file: 


>» cPickle. dump(a, open( ' my file, pickle' , 'wb')) 
>» c = cPickle. load(open( 'my file, pickle' , 'rb')) 


3 
Overview 


3.1 Startup 

web2py comes in binary packages for Windows and Mac OS X. They include 
the Python interpreter so you do not need to have it pre-installed. There 
is also a source code version that runs on Windows, Mac, Linux, and other 
Unix systems. The Windows and OS X binary versions include the necessary 
Python interpreter. The source code package assumes that Python is already 
installed on the computer. web2py requires no installation. To get started, 
unzip the downloaded zip file for your specific operating system and execute 
the corresponding web2py file. 

On Windows, run: 
web2py.exe 

On OS X, run: 

open web2py.app 

On Unix and Linux, run from source by typing: 

python2.5 web2py.py 

To run web2py on Windows from source install first Mark Hammond's "Python for 

Windows extensions, then run: 


70 WEB2PY FULL-STACK WEB FRAMEWORK, 4TH EDITION 


python2.5 web2py.py 


The web2py program accepts various command line options which are 
discussed later. 

By default, at startup, web2py displays a startup window and then displays 
a GUI widget that asks you to choose a one-time administrator password, 
the IP address of the network interface to be used for the web server, and a 
port number from which to serve requests. By default, web2py runs its web 
server on 127.0.0.1:8000 (port 8000 on localhost), but you can run it on any 
available IP address and port. You can query the IP address of your network 
interface by opening a command line and typing ipconfig on Windows or 
if config on OS X and Linux. From now on we assume web2py is running 
on localhost (127.0.0.1:8000). Use 0.0.0.0:80 to run web2py publicly on any of 
your network interfaces. 


web2p/ 


Welcome to... 

web2py Web Framework 

Created by Massimo Di Pierro, Copyright 2007-201 1 
Version 1.99.3 (2011-11-21 23:29:32) dev 


If you do not provide an administrator password, the administration interface 
is disabled. This is a security measure to prevent publicly exposing the admin 
interface. 

The administrative interface, admin, is only accessible from localhost unless 
you run web2py behind Apache with mod_proxy If admin detects a proxy, 
the session cookie is set to secure and admin login does not work unless the 
communication between the client and the proxy goes over HTTPS; this is a 
security measure. All communications between the client and admin must 
always be local or encrypted; otherwise an attacker would be able to perform 
a man-in-the middle attack or a replay attack and execute arbitrary code on 
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the server. 

After the administration password has been set, web2py starts up the web 
browser at the page: 

http://127. 0.0. 1:8000/ 


If the computer does not have a default browser, open a web browser and 
enter the URL. 



ttpV/ia7 0i-MroJwricBme/default/lndat 


; 





Welcome 




Mac 


O Share 

Hello World 

How did you gel here? 

1. You art auccesafully running vrab2py 

«e323yapslca;:onaWBln>me,'contr 
odrjylhoviov. 

_ — 

Don't know what b do? 

. webSpy.i^n 


Copyright £20.11 


■T^B^n 




Clicking on "administrative interface" takes you to the login page for the 
administration interface. 





r Login 

' -^ '' 

q http://1 27.0.0. t.SOOO/admiri/defaiilt/iridex 


»IC 




•WEB2PY™ WEB FRAMEWORK 

O Login to the Administrative Interface 





ATTENTION: Login require? a seco/e (H7TPS) connection or running c 

n idcalfloBt, 



Administrator Password: || 

[Latfnj 



The administrator password is the password you chose at startup. Notice 
that there is only one administrator, and therefore only one administrator 
password. For security reasons, the developer is asked to choose a new 
password every time web2py starts unless the <recycle> option is specified. 
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This is distinct from the authentication mechanism in web2py applications. 

After the administrator logs into web2py, the browser is redirected to the 
"site" page. 


CJ I jj hctp://iZ7D.a.iLEfloi)/admln/defau(t/sit 


~*l c *«>** 


> 


O Now application wizard 


0=i 


U ->'\ l-v "ii 'jT'O-II 1 ? '".p|: bnijn^ 


This page lists all installed web2py applications and allows the administrator 
to manage them. web2py comes with three applications: 

• An admin application, the one you are using right now. 

• An examples application, with the online interactive documentation and 
a replica of the web2py official website. 

• A welcome application. This is the basic template for any other web2py 
application. It is referred to as the scaffolding application. This is also the 
application that welcomes a user at startup. 

Ready-to-use web2py applications are referred to as web2py appliances. You 
can download many freely available appliances from [34]. web2py users are 
encouraged to submit new appliances, either in open-source or closed-source 
(compiled and packed) form. 

From the admin application's site page, you can perform the following 
operations: 

• install an application by completing the form on the bottom right of the 
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page. Give a name to the application, select the file containing a packaged 
application or the URL where the application is located, and click "submit". 

• uninstall an application by clicking the corresponding button. There is a 
confirmation page. 

• create a new application by choosing a name and clicking "create". 

• package an application for distribution by clicking on the corresponding 
button. A downloaded application is a tar file containing everything, 
including the database. You should not untar this file; it is automatically 
unpackaged by web2py when installed with admin. 

• clean up an application's temporary files, such as sessions, errors and 
cache files. 

• EDIT an application. 

When you create a new application using admin, it starts as a clone of 
the "welcome" scaffolding app with a " models /db.py" that creates a SQLite 
database, connects to it, instantiates Auth, Crud and Service, configures 
them. It also provides a "controller /default. py" which exposes actions "index", 
"download", "user" for user management, and "call" for services. In the 
following, we assume that these files have been removed; we will be creating 
appsfrom scratch. 

webzpy also comes with a wizard, described later in this chapter, that can write 
an alternate scaffolding code for you based on layouts and plugins available on 
the web and based on high level description of the models. 


3.2 Say hello 

Here, as an example, we create a simple web app that displays the message 
"Hello from My App" to the user. We will call this application "myapp". We 
will also add a counter that counts how many times the same user visits the 
page. 

You can create a new application simply by typing its name in the form on 
the top right of the site page in admin. 
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ONew simple application 

Application name: [myapp| 


Create 


After you press [create], the application is created as a copy of the built-in 
welcome application. 


EJ, I _J hctp://lZ7.0.a.l:B00D/admin/derauiydesign^myapp 


• EDIT APPLICATION "MYAPP" 


Ed-! P. 


O Controllers St 




LjgL 1 _mi'_.py 

^J g d 0(aU | U lnd=..M 
gUiJ g ,K1 aU IVL, aE ,.h<n 

ta»j 5 „».,i,„,,,„ 

OB F, g...n.,..". 
OJB I J....1. >... 


To run the new application, visit: 

http : //127 . . . 1 : 8000/myapp 


Now you have a copy of the welcome application. 

To edit an application, click on the design button for the newly created 
application. 

The Edit page tells you what is inside the application. Every web2py 
application consists of certain files, most of which fall into one of isx 
categories: 

• models: describe the data representation. 

• controllers: describe the application logic and workflow. 
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• views: describe the data presentation. 

• languages: describe how to translate the application presentation to other 
languages. 

• modules: Python modules that belong to the application. 

• static files: static images, CSS files [40, 41, 42], JavaScript files [43, 44], etc. 

• plugins: groups of files designed to work together. 

Everything is neatly organized following the Model- View-Controller design 
pattern. Each section in the edit page corresponds to a subfolder in the 
application folder. 

Notice that section headings will toggle their content. Folder names under 
static files are also collapsible. 

Each file listed in the section corresponds to a file physically located in the 
subfolder. Any operation performed on a file via the admin interface (create, 
edit, delete) can be performed directly from the shell using your favorite editor. 

The application contains other types of files (database, session files, error 
files, etc.), but they are not listed on the edit page because they are not 
created or modified by the administrator; they are created and modified by 
the application itself. 

The controllers contain the logic and workflow of the application. Every 
URL gets mapped into a call to one of the functions in the controllers 
(actions). There are two default controllers: "appadmin.py" and "default. py". 
appadmin provides the database administrative interface; we do not need 
it now. "default. py" is the controller that you need to edit, the one that is 
called by default when no controller is specified in the URL. Edit the "index" 
function as follows: 


def index( ) : 

return "Hello from MyApp" 


Here is what the online editor looks like: 
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q|jhtt P ://l?7 0.D.l:B000/^:„,i„/J-r,u:t/edfi/M,^p [ ,/c U „i i ulie,Vdef aU lLpj 


• EDITING F LET'' :■ R3/DEF/V 


»ie «■— ■ 


«oii|.i -' hin^if -'-" anln 




Save it and go back to the edit page. Click on the index link to visit the newly 
created page. 

When you visit the URL 

http : //127 . . . 1 : 8000/myapp/def ault/index 

the index action in the default controller of the myapp application is called. 
It returns a string that the browser displays for us. It should look like this: 


http://12T.0.0.1:8000/myapp/default/index 


!Z^i |LJ http://127.0.Q.1:BOOO/myapp/deFault/mdex 


Hello from MyApp 


Now, edit the "index" function as follows: 

def index( ) : 

return diet (message^" Hello from MyApp") 


Also from the edit page, edit the view "default/indexhtml" (the view file 
associated with the action) and completely replace the existing contents of 
that file with the following: 


1 <html> 

2 <headx/head> 
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<body> 

4 <hl>{{=message}}</hl> 

5 </body> 

6 </html> 


Now the action returns a dictionary defining a message. When an action 
returns a dictionary, web2py looks for a view with the name 

[controller]/[function] . [extension] and executes it. Here [extension] is the 
requested extension. If no extension is specified, it defaults to "html", and 
that is what we will assume here. Under this assumption, the view is an 
HTML file that embeds Python code using special jj }} tags. In particular, 
in the example, the {{=message}} instructs web2py to replace the tagged code 
with the value of the message returned by the action. Notice that message here 
is not a web2py keyword but is defined in the action. So far we have not used 
any web2py keywords. 

If web2py does not find the requested view, it uses the "generic.html" view 
that comes with every application. 

If an extension other than "html" is specified ("json" for example), and the 
view file "[controller] '/[function], json" is not found, webzpy looks for the view 
"generic. json". webzpy comes with generic.html, generic.json, generic.xml, 
and generic.rss. These generic views can be modified for each application 
individually, and additional views can be added easily. 

Generic views are a development tool. In production every action should have 
its own view. In fact, by default, generic views are only enabled from localhost. 

You can also specify a view with response. view = 'default/something, html' 

Read more on this topic in Chapter 10. 

If you go back to "EDIT" and click on index, you will now see the following 
HTML page: 


*» http://127.0.0.1:8000/myapp/default/index 


E^J, \\±l http://127.0.0.1:S000/niyapp/deFault/index 


Hello from MyApp 
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For debugging purposes you can always append 

{{= response. toolbar ( ) }} 

to the code in a view and it will show you some useful information, including 
the request, response and session objects, and list all db queries with their 
timing. 


3.3 Let's count 

Let's now add a counter to this page that will count how many times the 
same visitor displays the page. web2py automatically and transparently 
tracks visitors using sessions and cookies. For each new visitor, it creates 
a session and assigns a unique "session_id". The session is a container for 
variables that are stored server-side. The unique id is sent to the browser via 
a cookie. When the visitor requests another page from the same application, 
the browser sends the cookie back, it is retrieved by web2py, and the 
corresponding session is restored. 

To use the session, modify the default controller: 



Notice that counter is not a web2py keyword but session is. We are asking 
web2py to check whether there is a counter variable in the session and, if not, 
to create one and set it to 1. If the counter is there, we ask web2py to increase 
the counter by 1. Finally we pass the value of the counter to the view. 

A more compact way to code the same function is this: 

1 def index( ) : 

z session. counter = (session. counter or 0) + 1 

3 return diet (message="Hello from MyApp" , counter=session. counter) 

Now modify the view to add a line that displays the value of the counter: 
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i <html> 

2 <headx/head> 

i <body> 

4 <hl>{{=message}}</hl> 

-, <h2>Number of visits: {{=counter}}</h2> 

6 </body> 

7 </html> 


When you visit the index page again (and again) you should get the following 
HTML page: 


http://127.0.0.1:B000/myapp/defaulc/index 


d |lli httpy/IZ/.O.O.IiSOOO/myapp/default/i ndex % \ Q ^ Back @ 

Hello from MyApp 

Number of visits: 5 


The counter is associated with each visitor, and is incremented each time the 
visitor reloads the page. Different visitors see different counters. 

3.4 Say my name 


Now create two pages (first and second), where the first page creates a form, 
asks the visitor's name, and redirects to the second page, which greets the 
visitor by name. 



pffv; 



Write the corresponding actions in the default controller: 


def 

first(): 



return diet ( ) 

def 

second( ) 



return diet ( ) 
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Then create a view "default/first.html" for the first action, and enter: 



Finally, create a view "default/secondhtml" for the second action: 


{{extend ' layout. html ' }} 

<hl>Hello {{= request .vars .visitor_name}}</hl> 


In both views we have extended the basic "layout.html" view that comes with 
web2py. The layout view keeps the look and feel of the two pages coherent. 
The layout file can be edited and replaced easily, since it mainly contains 
HTML code. 

If you now visit the first page, type your name: 



L^ http://1 27.0.0. 1:8000/myapp/rJefault/f if st 

%\ Q ^Back i@ 

!_iide5iginmyapp ■ Myapp * | 

Myapp 

customize me! 

First 

Q Share 

Whai is your name? 


JBruno Submit 

Copyright ©2011 


hT^Pf^/l 



and submit the form, you will receive a greeting: 
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L^ http://T27.0.0.1:8000/myapp/del = dult/secorcl?visJtor_rame=Bruno 
Lijdesigirimyapp ■ Myapp * 


ll:J.!.IJ),!Jll.yi.UJU,.,.M 


Myapp 


customize me! 
Second 


Hello Bruno 


3.5 Postbacks 

The mechanism for form submission that we used before is very common, but 
it is not good programming practice. All input should be validated and, in 
the above example, the burden of validation would fall on the second action. 
Thus the action that performs the validation is different from the action that 
generated the form. This tends to cause redundancy in the code. 

A better pattern for form submission is to submit forms to the same action 
that generated them, in our example the "first". The "first" action should 
receive the variables, process them, store them server-side, and redirect the 
visitor to the "second" page, which retrieves the variables. This mechanism 
is called a postback. 



rtd 


Modify the default controller to implement self-submission: 



def first() 
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2 if request .vars.visitor_name: 

3 session. visitor_name = request .vars . visitor_name 

4 redirect(URL( 'second' ) ) 

5 return diet ( ) 

6 

7 def second( ) : 

s return diet ( ) 

Then modify the "default/ first.html" view: 

1 {{extend 'layout.html'}} 

2 What is your name? 

3 <form> 

4 <input name='Visiror_name" /> 

5 <input type="submit" /> 

6 </form> 


and the "default/second. html" view needs to retrieve the data from the 
session instead of from the request .vars: 


1 {{extend 'layout.html'}} 

2 <hl>Hello {{=session.visitor_name or "anonymous"}}</hl> 

From the point of view of the visitor, the self -submission behaves exactly the 
same as the previous implementation. We have not added validation yet, but 
it is now clear that validation should be performed by the first action. 

This approach is better also because the name of the visitor stays in the 
session, and can be accessed by all actions and views in the applications 
without having to be passed around explicitly. 

Note that if the "second" action is ever called before a visitor name is set, it 
will display "Hello anonymous" because session. visitor_name returns None. 
Alternatively we could have added the following code in the controller 
(inside the second function): 

1 if not request .function==' first ' and not session. visitor_name: 
redirect(URL(' first')) 

This is a general mechanism that you can use to enforce authorization on 
controllers, though see Chapter 9 for a more powerful method. 

With web2py we can move one step further and ask web2py to generate the 
form for us, including validation. web2py provides helpers (FORM, INPUT, 
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TEXTAREA, and SELECT/OPTION) with the same names as the equivalent 
HTML tags. They can be used to build forms either in the controller or in the 
view. 

For example, here is one possible way to rewrite the first action: 

def first() : 

form = FORM(INPUT(_name='i/isitor_name' , requires=IS_NOT_EMPTY( ) ) , 

INPUT(_type='suomit' )) 
if form. process! ) .accepted: 

session. visitor_name = form.vars.visitor_name 
redirect (URL( 'second' ) ) 
return diet (form=form) 


where we are saying that the FORM tag contains two INPUT tags. The 
attributes of the input tags are specified by the named arguments starting 
with underscore. The requires argument is not a tag attribute (because it does 
not start by underscore) but it sets a validator for the value of visitor_name. 

Here is yet another better wat to create the same form: 


def 

first() : 


form = SQLFORM. facto ry( Field ( 'visitor-name' , requires=IS_NOT_EMPTY( ) ) ) 


if form. process! ) .accepted : 


session. visitor_name = form.vars.visitor_name 


redirect (URL( 'second' ) ) 


return diet (form=form) 


The form object can be easily serialized in HTML by embedding it in the 
"default/ first.html" view. 

1 {{extend 'layout.html'}} 

2 What is your name? 

3 {{=form}} 


The form, process ( ) method applies the validators and returns the form itself. 
The f o rm . accepted variable is set to True if the form was processed and passed 
validation. If the self-submitted form passes validation, it stores the variables 
in the session and redirects as before. If the form does not pass validation, 
error messages are inserted into the form and shown to the user, as below: 
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L^ http://1 27.0.0. liaOOtymyapp/deFauiy first 

»ie «-n a ck 

,'^designmyapp H Myapp ■] 

Myapp 

customize me! 



First 

Q Share 

Whai is your name? 


III^^^^^^^M 

Submit 

Copyright©2011 

ras=B>isvH 



In the next section we will show how forms can be generated automatically 
from a model. 


3.6 An image blog 


Here, as another example, we wish to create a web application that allows the 
administrator to post images and give them a name, and allows the visitors 
of the web site to view the named images and submit comments. 

As before, from the site page in admin, create a new application called images, 
and navigate to the edit page: 
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:ttp./ ; i2- ■) 1 iiiOH/admin/default/design/images 


► EDIT APPLICATION "IMAGES" 




O Controllers © 

she! [eai QUuLaU 

Ligflj P. ^, r.ppPd.ln 
LiaU P. % dcfeulLpy 




-^Y^TT^Y; 


We start by creating a model, a representation of the persistent data in the 
application (the images to upload, their names, and the comments). First, 
you need to create /edit a model file which, for lack of imagination, we call 
"db.py". We assume the code below will replace any existing code in "db.py". 
Models and controllers must have a . py extension since they are Python code. 
If the extension is not provided, it is appended by web2py. Views instead 
have a .html extension since they mainly contain HTML code. 

Edit the "db.py" file by clicking the corresponding "edit" button: 



http://l 27.0.0. 1i:8000/admin/defaijlt/de5igii /images 
' Site ^ Edit 


J ABoiit If Errors If Versioning If Logout If help IB 


I K <">-py 

) g menu.py 


http://1 27.0.0. 1 :8000/admin/default/edk/images/modeb/db.py 


and enter the following: 


db 

= DAL( 

'sqlite 

■//storage. sqli 

te") 

db 

define. 

.tablet 

image ' , 




Field ( 

title' 

unique 

=True) , 
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5 Field( ' file' , 'upload'), 

6 format = '%(title)s' ) 
7 

8 db.define_table( ' comment' , 

Field ( ' image__id' , db. image), 
[o Field ( 'author' ) , 
1: Field ( 'email ' ) , 

Field ( 'body' , 'text')) 
13 

14 db. image. title. requires = IS_NOT_IN_DB(db, db. image. title) 

[ 5 db. comment . image_id. requires = IS_IN_DB(db, db. image. id, '%(title,)s' 

16 db. comment .author. requires = IS_NOT_EMPTY( ) 

17 db. comment .email. requires = IS_EMAIL() 

18 db. comment . body . requires = IS_NOT_EMPTY( ) 
iy 

21) db. comment . image_id. writable = db. comment . image_id. readable = False 


Let's analyze this line by line. 

Line 1 defines a global variable called db that represents the database 
connection. In this case it is a connection to a SQLite database stored in 
the file "applications/images/databases/storage. sqlite". In the SQLite case, 
if the database does not exist, it is created. You can change the name of the 
file, as well as the name of the global variable db, but it is convenient to give 
them the same name, to make it easy to remember. 

Lines 3-5 define a table "image". define_table is a method of the db object. 
The first argument, "image", is the name of the table we are defining. The 
other arguments are the fields belonging to that table. This table has a field 
called "title", a field called "file", and a field called "id" that serves as the table 
primary key ("id" is not explicitly declared because all tables have an id field 
by default). The field "title" is a string, and the field "file" is of type "upload", 
"upload" is a special type of field used by the web2py Data Abstraction Layer 
(DAL) to store the names of uploaded files. web2py knows how to upload 
files (via streaming if they are large), rename them safely, and store them. 

When a table is defined, web2py takes one of several possible actions: 

• if the table does not exist, the table is created; 

• if the table exists and does not correspond to the definition, the table is 
altered accordingly, and if a field has a different type, web2py tries to 
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convert its contents; 

• if the table exists and corresponds to the definition, web2py does nothing. 

This behavior is called "migration". In web2py migrations are automatic, but 
can be disabled for each table by passing migrate=False as the last argument 
of def ine_table. 

Line 6 defines a format string for the table. It determines how a record 
should be represented as a string. Notice that the format argument can also 
be a function that takes a record and returns a string. For example: 

format=lambda row: row. title 

Lines 8-12 define another table called "comment". A comment has an 
"author", an "email" (we intend to store the email address of the author of 
the comment), a "body" of type "text" (we intend to use it to store the actual 
comment posted by the author), and an "image_id" field of type reference 
that points to db. image via the "id" field. 

In line 14, db. image. title represents the field "title" of table "image". The 
attribute requires allows you to set requirements /constraints that will be 
enforced by web2py forms. Here we require that the "title" is unique: 

IS_NOT_IN_DB(db, db. image. title) 

Notice this is optional because it is set automatically given that Field ( 'title' , 
unique=True). 

The objects representing these constraints are called validators. Multiple 
validators can be grouped in a list. Validators are executed in the order they 
appear. IS_NOT_IN_DB(a, b) is a special validator that checks that the value of 
a field b for a new record is not already in a. 

Line 15 requires that the field "image_id" of table "comment" is in db . image . id. 
As far as the database is concerned, we had already declared this when we 
defined the table "comment". Now we are explicitly telling the model that 
this condition should be enforced by web2py, too, at the form processing 
level when a new comment is posted, so that invalid values do not propagate 
from input forms to the database. We also require that the "image_id" be 
represented by the "title", '%( title ) s ', of the corresponding record. 
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Line 20 indicates that the field "image_id" of table "comment" should 
not be shown in forms, writable=False and not even in readonly forms, 
readable=False. 

The meaning of the validators in lines 15-17 should be obvious. 

Notice that the validator 

db. comment. image_id. requires = IS_IN_DB(db, db. image. id, '%(title)s' ) 


can be omitted (and would be automatic) if we specify a format for referenced 
table: 


db.define_table( 'image' 


format='%(title)s' 


where the format can be a string or a function that takes a record and returns 
a string. 

Once a model is defined, if there are no errors, web2py creates an application 
administration interface to manage the database. You access it via the 
"database administration" link in the edit page or directly: 

http ://127 .0.0 . 1: 8000/images/appadmin 


Here is a screenshot of the app admin interface: 
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This interface is coded in the controller called "appadmin.py" and the 
corresponding view "appadmin.html". From now on, we will refer to this 
interface simply as appadmin. It allows the administrator to insert new 
database records, edit and delete existing records, browse tables, and perform 
database joins. 

The first time appadmin is accessed, the model is executed and the tables are 
created. The web2py DAL translates Python code into SQL statements that 
are specific to the selected database back-end (SQLite in this example). You 
can see the generated SQL from the edit page by clicking on the "sql.log" link 
under "models". Notice that the link is not present until the tables have been 
created. 



• PEEKING AT FILE "IMAGES/DAT ABASES/SQL. LOG" 


timestamp: 2011-11-23710:56:40.054532 
CREATE TABLE image* 

id INTEGER PRIMARY KEY AUTOINCREMENT, 

title CHAR1512] UNIQUE, 

file CHARI512J 


timestame: 2011-11-23110:56:40.218755 
CREATE TABLE comment; 

id INTEGER PRIMARY KEY AUTCiINCREMENT, 

image_id INTEGER REFERENCES image(id) ON DELETE CASCADEj 

author CHAR( 512] , 

email CHARI 512J , 
ody TEXT 


.'; 


If you were to edit the model and access appadmin again, web2py would 
generate SQL to alter the existing tables. The generated SQL is logged into 
"sql.log". 

Now go back to appadmin and try to insert a new image record: 
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}_A hitp://127.O.(}.1:8O00/images/appadmJr/irseft/db/imdge 

* ie 

^- Back 2- 

Images 

customize me! 



Insert 


O Share 

database db table 

New Record 

image 


Title: Image of tlie earth 


File: [choose File ■ earth.jpq 
Submit 

C<3pyright©2Qt1 
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web2py has translated the db. image. file "upload" field into an upload form 
for the file. When the form is submitted and an image file is uploaded, the 
file is renamed in a secure way that preserves the extension, it is saved with 
the new name under the application "uploads" folder, and the new name is 
stored in the db . image . file field. This process is designed to prevent directory 
traversal attacks. 

Notice that each field type is rendered by a widget. Default widgets can be 
overridden. 

When you click on a table name in appadmin, web2py performs a select of 
all records on the current table, identified by the DAL query 


db. image. id > 


and renders the result. 
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http://127.0.0.1:8000/imagei/appadmirysele.:t:/db?query=db. image. id=-0 *■ j= Back 


Update: [ 






The "query" is a condition like "db.Eable 1 .fieldl == , value"'. Something like "db.tablel .field 1 ==db.table2.f eld2" n 

a SQL JOIN. 

Use (...)&(...) for AND, (...)l(-) for OR, and -(...] for NOT to build more complex queries. 

"update" is an optional expression like "fieldl ='newvalue"\ Yoj cannot update or delete the results of a JOIN 


1 selected 


image, idimage.title imag 

1 Image ef the ...file 


Import/Export 


r import from osvfile | Choose File | (None) 


You can select a different set of records by editing the SQL query and pressing 
[Submit] . 

To edit or delete a single record, click on the record id number. 

Because of the IS_IN_DB validator, the reference field "image_id" is rendered 
by a drop-down menu. The items in the drop-down are stored as keys 
(db. image. id), but are represented by their db. image. title, as specified by 
the validator. 

Validators are powerful objects that know how to represent fields, filter field 
values, generate errors, and format values extracted from the field. 

The following figure shows what happens when you submit a form that does 
not pass validation: 
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L^ http://1 27.0.0.1 tfOOO/images/appadmin/insert/db/comment %\ Q ^ Back (S) 

database db table comment 

New Record 
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Author: 
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Email: 



Body: 




Submit 



The same forms that are automatically generated by app admin can also be 
generated programmatically via the SQLFORM helper and embedded in user 
applications. These forms are CSS-friendly and can be customized. 

Every application has its own app admin; therefore, app admin itself can be 
modified without affecting other applications. 

So far, the application knows how to store data, and we have seen how to 
access the database via appadmin. Access to appadmin is restricted to the 
administrator, and it is not intended as a production web interface for the 
application; hence the next part of this walk-through. Specifically we want to 
create: 

• An "index" page that lists all available images sorted by title and links to 
detail pages for the images. 

• A "show/[id]" page that shows the visitor the requested image and allows 
the visitor to view and post comments. 

• A "download/[name]" action to download uploaded images. 
This is represented schematically here: 
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Go back to the ecfif P a ge and edit the "default.py" controller, replacing its 
contents with the following: 


def index( ) : 


images = db( ). select(db. image. ALL, 

orderby=db. image. title) 

return diet (images=images) 



This action returns a dictionary. The keys of the items in the dictionary are 
interpreted as variables passed to the view associated to the action. When 
developing, if there is no view, the action is rendered by the "generic.html" 
view that is provided with every web2py application. 

The index action performs a select of all fields (db. image. ALL) from table 
image, ordered by db. image. title. The result of the select is a Rows object 
containing the records. Assign it to a local variable called images returned by 
the action to the view, images is iterable and its elements are the selected rows. 
For each row the columns can be accessed as dictionaries: images [0] [ 'title' ] 
or equivalently as images [0] .title. 

If you do not write a view, the dictionary is rendered by "views/generic. html" 
and a call to the index action would look like this: 


. : = each 


Images 

customize me! 


lma 9 es image. idimage.title image.lil 


Administrative Interface 

Don'lknowwhallado? 

• Online examples 
. web2py.com 

• Documentation 


You have not created a view for this action yet, so web2py renders the set of 
records in plain tabular form. 

Proceed to create a view for the index action. Return to admin, edit 
"default/ index. html" and replace its content with the following: 
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The first thing to notice is that a view is pure HTML with special ((...}} tags. 
The code embedded in ({...}} is pure Python code with one caveat: indentation 
is irrelevant. Blocks of code start with lines ending in colon (:) and end in 
lines beginning with the keyword pass. In some cases the end of a block is 
obvious from context and the use of pass is not required. 

Lines 5-7 loop over the image rows and for each row image display: 

LI(A(image. title, _href=URL( ' show' , args=image.id) ) 

This is a <li>. . .</li> tag that contains an <a href=" ...">.. .</a> tag which 
contains the image, title. The value of the hypertext reference (href attribute) 

is: 

URL( 'show', args=image.id) 

i.e., the URL within the same application and controller as the current 
request that calls the function called "show", passing a single argument to 
the function, args=image.id. LI, A, etc. are web2py helpers that map to 
the corresponding HTML tags. Their unnamed arguments are interpreted 
as objects to be serialized and inserted in the tag's innerHTML. Named 
arguments starting with an underscore (for example _href) are interpreted 
as tag attributes but without the underscore. For example _href is the href 
attribute, _ class is the class attribute, etc. 

As an example, the following statement: 

{{=LI ( A( 'something' , _href=URI_( 'show' , args=123))}} 

is rendered as: 

<lixa href="/images/£/efauIt/s/)ow/123">something</a></li> 

A handful of helpers (INPUT, TEXTAREA, OPTION and SELECT) also support some 
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special named attributes not starting with underscore (value, and requires). 
They are important for building custom forms and will be discussed later. 

Go back to the edit page. It now indicates that "default.py exposes index". By 
clicking on "index", you can visit the newly created page: 
http : //127 . . . 1 : 8000/images/def ault/index 


which looks like: 


-ictp:/7 1 l : o. j i :ooi)o/lmages/ 


_gc *■« 


Images 


Current Images 


If you click on the image name link, you are directed to: 

http : //127 . . . 1 : 8000/images/def ault/show/1 


and this results in an error, since you have not yet created an action called 
"show" in controller "default.py". 

Let's edit the "default.py" controller and replace its content with: 

def index( ) : 

images = db() .select (db. image. ALL, orderby=db. image. title) 
return diet (images=images) 

def show( ) : 

image = db(db. image. id==request. args(0) ). select( ) .first( ) 
db. comment .image_id. default = image. id 
form = SQLFORM(db. comment) 
if form. process! ) .accepted: 

response. flash = 'your comment is posted' 
comments = db(db. comment .image_id==image. id) . select( ) 
return diet (image=image, comments=comments, form=form) 

def download) ) : 

return response. download (request, db) 
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The controller contains two actions: "show" and "download". The "show" 
action selects the image with the id parsed from the request args and all 
comments related to the image, "show" then passes everything to the view 
"default/ showhtml" . 

The image id referenced by: 

URL( 'show', args=image.id) 

in "default/index. html", can be accessed as: request .args (0) from the "show" 
action. 

The "download" action expects a filename in request. args(0), builds a path 
to the location where that file is supposed to be, and sends it back to the 
client. If the file is too large, it streams the file without incurring any memory 
overhead. 

Notice the following statements: 

• Line 7 creates an insert form SQLFORM for the db . comment table using only 
the specified fields. 

• Line 8 sets the value for the reference field, which is not part of the input 
form because it is not in the list of fields specified above. 

• Line 9 processes the submitted form (the submitted form variables are in 
request .vars) within the current session (the session is used to prevent 
double submissions, and to enforce navigation). If the submitted form 
variables are validated, the new comment is inserted in the db. comment 
table; otherwise the form is modified to include error messages (for 
example, if the author's email address is invalid). This is all done in line 
9!. 

• Line 10 is only executed if the form is accepted, after the record is 
inserted into the database table, response. flash is a web2py variable that 
is displayed in the views and used to notify the visitor that something 
happened. 

• Line 11 selects all comments that reference the current image. 

The "download" action is already defined in the " default. py" controller of the 
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scaffolding application. 

The "download" action does not return a dictionary, so it does not need a 
view. The "show" action, though, should have a view, so return to admin 
and create a new view called "default/ show.html". 

Edit this new file and replace its content with the following: 


] {{extend ' layout. html ' }} 

z <hl>Image: {{=image.title}}</hl> 

3 <center> 

4 <img width="200px" 

5 src=" {{=URL( ' download' , args=image.file)}}" /> 

6 </center> 

7 {{if ten ( comments ) :}} 

8 <h2>Comments</h2xbr /><p> 

i) {{for comment in comments:}} 

io <p>{{=comment .author}} says <i>{{=comment . body}}</ix/p> 

{{pass}}</p> 

,z {{else:}} 

n <h2>No comments posted yet</h2> 

i 4 {{pass}} 

I5 <h2>Post a comment</h2> 

in {{=form}} 


This view displays the image.file by calling the "download" action inside an 
<img ... /> tag. If there are comments, it loops over them and displays each 
one. 

Here is how everything will appear to a visitor. 
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When a visitor submits a comment via this page, the comment is stored in 
the database and appended at the bottom of the page. 

3.7 Adding CRUD 

web2py also provides a CRUD (Create/Read/Update/Delete) API that 
simplifies forms even more. To use CRUD it is necessary to define it 
somewhere, such as in file "db.py": 

from gluon. tools import Crud 
crud = Crud(db) 

These two lines are already in the scaffolding application. 

The crud object provides high-level methods, for example: 

form = crud. create(table) 


that can be used to replace the programming pattern: 


, form = SQLFORH( table) 

2 if form. process) ) .accepted: 
1 session. flash = ' . . . ' 

4 redirect ( '...') 
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Here, we rewrite the previous "show" action using crud and making some 
more improvements: 



First of all notice we have used the syntax 
db. image ( request. a rgs (0) ) or redirect ( . . . ) 

to fetch the required record. Since 'table (id) returns None if the record is 
not found, we can use or redirect (...) in this case in one line. 

The next argument of crud. create is the URL to redirect to after the form is 
accepted. The message argument is the one to be displayed upon acceptance. 
You can read more about CRUD in Chapter 7. 


3.8 Adding Authentication 

The web2py API for Role-Based Access Control is quite sophisticated, but 
for now we will limit ourselves to restricting access to the show action to 
authenticated users, deferring a more detailed discussion to Chapter 9. 

To limit access to authenticated users, we need to complete three steps. In a 
model, for example "db.py", we need to add: 

1 from gluon. tools import Auth 

2 auth = Auth(db) 

3 auth.define_tables( ) 

In our controller, we need to add one action: 

1 def user() : 

2 return diet (form=auth( ) ) 
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This is sufficient to enable login, register, logout, etc. pages. The default 
layout will also show options to the corresponding pages in the top right 


corner. 
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We can now decorate the functions that we want to restrict, for example: 



Any attempt to access 


http : //127 . . . 1 : 8000/images/def ault/show/ [image_id] 


will require login. If the user is not logged it, the user will be redirected to 


http : //127 . . . 1 : 8000/images/def ault/user/login 
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The user function also exposes, among others, the following actions: 


http://127.0.O.l:8000/images/default/user/logout 
http://127.0.0.1:8000/images/default/user/ register 
http : //127 . . . 1 : 8000/images/def ault/user/p rof ile 
http://127.0.0. 1: 8000/images/def ault/user/change_password 
http:// 127. 0.0. 1: 8000/images/def ault/user/ request. reset. pas sword 
http : //127 .0.0.1: 8000/images/def ault/user/ ret rieve_username 
http : //127 .0.0.1: 8000/images/def ault/user/ ret rieve_ pas sword 
http ://127. 0.0.1: 8000/images/def ault/user/verify_email 
http : //127 . . . 1 : 8000/images/def ault/user/impersonate 
http ://127. 0.0.1: 8000/images/def ault/user/not_authorized 


Now, a first-time user needs to register in order to be able to log in and read 
or post comments. 

Both the auth object and the user function are already defined in the scaffolding 
application. The auth object is highly customizable and can deal with email 
verification, registration approvals, CAPTCHA, and alternate login methods 
via plugins. 
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3.8.1 Adding grids 


We can improve this further using the SQLFORM.grid and SQLFORM.smartgrid 
gadgets to create a management interface for our application: 


@auth. requires_membership( 'manager' ) 
def manage ( ) : 

grid = SQLFORM.smartgrid(db. image) 

return diet (grid=grid) 


with associated "views /default /manage. html" 

{{extend 'layout.html'}} 
<h2>Management Interface</h2> 
{{=grid}} 


Using appadmin create a group "manager" and make some users members 
of the group. They will not be able to access 

http://127.0.0.1:8000/images/default/manage 


and browse, search: 


i_^ http://1?7 0.0 1:S00C/images/deFault/manage/image 
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create, update and delete images and their comments: 
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http://127.O.fM:BOO0/imagei/deFault/manage/image/edit/image/1?_sigrature=6t 
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3.9 Configuring the layout 

You can configure the default layout by editing "views/layout.html" but you 
can also configure it without editing the HTML. In fact, the "static/base. ess" 
stylesheet is well documented and described in Chapter 5. You can change 
color, columns, size, borders and background without editing the HTML. If 
you want to edit the menu, the title or the subtitle, you can do so in any 
model file. The scaffolding app, sets default values of these parameters in 
the file "models/menu. py": 


response. title = request .application 

response. subtitle = T( ' customize me/') 

response. meta. author = 'you' 

response. meta. description = 'describe your app' 

response. meta. keywords = 'bla bla bla' 

response. menu = [ [ 'Index', False, URL( 'index' ) ] ] 
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3.20 A wiki 

In this section, we build a wiki, from scratch and without using the extended 
functionality provided by plugin_wiki which is described in chapter 12. The 
visitor will be able to create pages, search them (by title), and edit them. 
The visitor will also be able to post comments (exactly as in the previous 
applications), and also post documents (as attachments to the pages) and 
link them from the pages. As a convention, we adopt the Markmin syntax 
for our wiki syntax. We will also implement a search page with Ajax, an RSS 
feed for the pages, and a handler to search the pages via XML-RPC [46]. 

The following diagram lists the actions that we need to implement and the 
links we intend to build among them. 
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Start by creating a new scaffolding app, naming it "mywiki". 

The model must contain three tables: page, comment, and document. Both 
comment and document reference page because they belong to page. A 
document contains a file field of type upload as in the previous images 
application. 

Here is the complete model: 

db = DAL( ' sqlite : //storage, sqlite' ) 

from gluon. tools import * 
auth = Auth(db) 
auth.define_tables( ) 
crud = Crud(db) 

s db.define_table( 'page' , 
, Field ('title'), 

Field ( ' body ' , ' text ' ) , 
u Field ( 'created^on ' , 'datetime', default=request .now) , 

12 Field ( 'created-by' , db.auth_user, default=auth.user_id) , 

13 format='%ftitZe,)s' ) 

[ 5 db.define_table( ' comment' , 

16 Field ( 'page^id' , db.page), 

.7 Field (' body ' , 'text'), 

18 Field ( 'created^on ' , 'datetime', default=request .now) , 

i 9 Field ( 'created-by' , db.auth_user, default=auth.user_id) ) 

21 db.define_table( 'document ' , 

22 Field ( 'page_id' , db.page), 

23 Field ( ' name ' ) , 

24 Field( ' file' , 'upload' ) , 

25 Field ( 'created-on ' , 'datetime', default=request .now) , 

26 Field ( 'created_by' , db.auth_user, default=auth.user_id) , 

27 fo rmat='% (name)s ' ) 
28 

29 db.page. title. requires = IS_NOT_IN_DB(db, ' page. title' ) 

30 db.page. body. requires = IS_NOT_EMPTY( ) 

31 db. page. created_by. readable = db. page. created_by. writable = False 

32 db. page. created_on. readable = db. page. created_on. writable = False 

33 

34 db. comment. body. requires = IS_NOT_EMPTY( ) 

35 db. comment . page_id. readable = db. comment. page_id. writable = False 

36 db. comment .created_by. readable = db. comment . created_by. writable = False 

37 db. comment . created_on. readable = db. comment . created_on. writable = False 


db. document . name. requires = IS_NOT_IN_DB(db, 'document .name' ) 
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40 db. document 

. page_id. readable = db. document . page_id. writable = False 

4 ] db. document 

. created_by. readable = db. document . created_by. writable = False 

42 db. document 

. created_on. readable = db. document . created_on. writable = False 


Edit the controller "default.py" and create the following actions: 

• index: list all wiki pages 

• create: post another wiki page 

• show: show a wiki page and its comments, and append comments 

• edit: edit an existing page 

• documents: manage the documents attached to a page 

• download: download a document (as in the images example) 

• search: display a search box and, via an Ajax callback, return all matching 
titles as the visitor types 

• callback: the Ajax callback function. It returns the HTML that gets 
embedded in the search page while the visitor types. 

Here is the "default.py" controller: 

1 def index( ) : 

2 this controller returns a dictionary rendered by the view 

3 it lists all wiki pages 

4 >» index() .has-keyf pages' ) 

5 True 

6 

7 pages = db() .select(db. page. id, db. page. title, orderby=db. page. title) 
s return diet (pages=pages) 

K. @auth. requires_login( ) 

u def create( ) : 

12 "creates a new empty wiki page" 

13 form = crud.create(db. page, next=URL( 'index' ) ) 

14 return dict(form=form) 

it, def show( ) : 

17 "shows a wiki page" 

18 this_page = db. page( request. a rgs (0) ) or redirect ( URL ( 'index' ) ) 
i 9 db. comment .page_id. default = this_page.id 

20 form = crud.create(db. comment) if auth.user else None 

21 pagecomments = db(db. comment .page_id==this_page. id) . select ( ) 

22 return diet (page=this_page, comments=pagecomments, form=form) 
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23 

24 

@air 

th. requires_login( ) 

25 

def 

edit() : 

26 


"edit an existing wiki page" 

27 


this_page = db.page(request.args(0)) or redirect ( URL ( 'index' ) ) 

28 


form = crud.update(db. page, this_page, 

29 


next=URL( 'show' ,args=request .args) ) 

30 


return dict(form=form) 

3 1 
32 

@air 

th. requires_login( ) 

33 

def 

documents ( ) : 

34 


"browser, edit all documents attached to a certain page" 

35 


page = db. page(request .args(0) ) or redirect (URL( 'index' ) ) 

36 


db. document .page_id. default = page. id 

37 


db. document .page_id. writable = False 

38 


grid = SQLFORM. g rid (db. document .page_id==page. id, a rgs=[ page. id] ) 

39 


return diet (page=page, grid=grid) 

40 
4 1 

def 

user( ) : 

42 


return dict(form=auth( ) ) 

43 

44 

def 

download ( ) : 

45 


"allows downloading of documents" 

46 


return response. download (request, db) 

47 
48 

def 

search( ) : 

49 


"an ajax wiki search page" 

50 


return diet (form=FORM(INPUT(_id=' keyword' ,_name=' keyword' , 

51 


_onkeyup="a_/axf 'callback' , ['keyword' ] , 'target' );")) , 

52 


target_div=DIV(_id=' target ' ) ) 

53 
54 

def 

callback!) : 

55 


"an ajax callback that returns a <ul> of links to wiki pages" 

56 


query = db. page. title. contains(request.vars. keyword) 

57 


pages = db(query) .select (orderby=db. page. title) 

58 


links = [A(p. title, _href=URL( 'show' ,args=p. id) ) for p in pages] 

59 


return UL(*links) 


Lines 2-6 provide a comment for the index action. Lines 4-5 inside the 
comment are interpreted by python as test code (doctest). Tests can be run 
via the admin interface. In this case the tests verify that the index action runs 
without errors. 

Lines 18, 27, and 35 try to fetch a page record with the id in request .args (0). 

Lines 13, 20 define and process create forms for a new page and a new 
comment and. 
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Line 28 defines and processes an update form for a wiki page. 

Line 38 creates a grid object that allows to browser, add and update the 
comments linked to a page. 

Some magic happens in line 51. The onkeyup attribute of the INPUT tag 
"keyword" is set. Every time the visitor releases a key, the JavaScript code 
inside the onkeyup attribute is executed, client-side. Here is the JavaScript 
code: 


aj ax (' callback' , [ 'keyword' ] , 'target'); 


ajax is a JavaScript function defined in the file "web2pyjs" which is included 
by the default "layout.html". It takes three parameters: the URL of the action 
that performs the synchronous callback, a list of the IDs of variables to be 
sent to the callback (["keyword"]), and the ID where the response has to be 
inserted ("target"). 

As soon as you type something in the search box and release a key, the 
client calls the server and sends the content of the 'keyword' field, and, 
when the sever responds, the response is embedded in the page itself as 
the innerHTML of the 'target' tag. 

The 'target' tag is a DIV defined in line 52. It could have been defined in the 
view as well. 

Here is the code for the view "default/create.html": 


1 {{extend 'layout.html'}} 

z <hl>Create new wiki page</hl> 

3 {{=form}} 


If you visit the create page, you see the following: 
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http://1Z7.0.0.1:8000/mywiki/default/create 


15 c ^ Bad( 


Mywiki 


customize me! 
Create 


Create new wiki page 


Title: My Main Paje 


Body: Hello today I made my first "|jvlkl" application 


Here is the code for the view "default/index.html": 



It generates the following page: 
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L^ httpV/^.a.O.I^OOO/mywiki/deFault/index 

* C? T* Ba(:k 

B^^^^^^^^^^^^^^^^^^^^^^Q 

Mywiki 

customize me! 

Index 

O Share 

Available wiki pages 

[ search ] 
• My Main Page 

[create page] 


Copyright© 2011 


l™-p"JS^/l 



Here is the code for the view "default/show.html": 

1 {{extend ' layout. html ' }} 

2 <hl>{{=page.title}}</hl> 

1 [ {{=A('edit' , _href=URL( 'edit' , args=request .args) )}} 

4 I {{=A( ' documents' , _href=URI_( ' documents' , args=request .args) )}} ]<br /> 

5 {{=MARKMIN(page.body)}} 

6 <h2>Comments</h2> 

- {{for comment in comments:}} 

8 <p>{{=db.auth_user[comment . created_by] . first_name}} on {{=comment .created_on}} 

9 says <I>{{=comment . body}}</ix/p> 
... {{pass}} 

11 <h2>Post a comment</h2> 

, 2 {{=form}} 


If you wish to use markdown syntax instead of markmin syntax: 
from gluon. contrib. markdown import WIKI 

and use WIKI instead of the MARKMIN helper. Alternatively you can choose to 
accept raw HTML instead of markmin syntax. In this case you would replace: 

{{=MARKMIN(page.body)}} 


with: 
{{=XML(page.body)}} 
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(so that the XML does not get escaped, as by default web2py behavior). 

This can be done better with: 

{{=XML(page. body, sanitize=True)}} 

By setting sanitize=True, you tell web2py to escape unsafe XML tags such as 
"<script>", and thus prevent XSS vulnerabilities. 

Now if, from the index page, you click on a page title, you can see the page 
that you have created: 



Mywiki 

customize me! 
Show 


My Main Page 

[edit | documents] 

Hello today I made my first wiki application 

Comments 

Bruno on 201 1-1 1-23 1 1:41 :55 says Hi there 


Post a comment 

Body: 


_A I C 4- b^ 


Q Share 


Here is the code for the view "default/edithtml": 

i {{extend ' layout. html ' }} 

2 <hl>Edit wiki page</hl> 

i [ {{=A('show', _href=URL( 'show' , args=request .args) )}} ]<br /> 

4 {{=form}} 


It generates a page that looks almost identical to the create page. 
Here is the code for the view "default/documents. html": 
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If, from the "show" page, you click on documents, you can now manage the 
documents attached to the page. 


l_^ http://1 27.0.0. T:8000/mywiki/tiefaiilt/dcKiimen 

/1 


»j C ^Back 

Mywiki 

customize me! 

Documents 



Osr,.r. 

Documents for 

Documents 

page: 

My Main Page 

query 

Search Clear 

Add Export 


Id Name File 

1 Image of earth F 

le View Edit Delete 



CopyrighS0 2Q11 



ivaiapyl 



Finally here is the code for the view "default/searchhtml" 


1 {{extend ' layout. html ' }} 

2 <hl>Search wiki pages</hl> 

3 [ {{=HCUstaW , _href=URL(' index' ))}}]<br /> 

4 {{=form}}<br />{{=target_div}} 


which generates the following Ajax search form: 
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L^ http://1 27.0.0. 1:800a/mywjkj/default/search 

■■;■ ■; ,p>Back 


Mywiki 

customize me! 

Search 

Q Share 

Search wiki pages 

[listall] 
|my| 

• My Main Page 

Copyright ©2011 


I^Mfcrl 



You can also try to call the callback action directly by visiting, for example, 

the following URL: 

http : //127 . . . 1 : 8000/mywiki/def ault/caUback?keyword=wiki 

If you look at the page source you see the HTML returned by the callback: 

<ulxlixa href="/itiywiki/default/show/4">l made a Wiki</ax/lix/ul> 


Generating an RSS feed from the stored pages using web2py is easy because 
web2py includes gluon.contrib. rss2. Just append the following action to the 
default controller: 

def news( ) : 

"generates rss feed form the wiki pages" 
reponse.generic_patterns = ['.rss'] 

pages = db( ). select(db. page. ALL, orderby=db. page. title) 
return dict( 

title = 'mywiki rss feed' , 

link = 'http://127.0.0.1:8696/mywiki/default/index' , 

description = 'mywiki news', 

created_on = request. now, 

items = [ 

dict(title = row. title, 

link = URL( 'show', args=row.id) , 
description = HARKMIN( row. body) .xml( ) , 
created_on = row.created_on 
) for row in pages] ) 
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and when you visit the page 
http://127.0.0.1:8000/mywiki/default/news. rss 


you see the feed (the exact output depends on the feed reader). Notice that 
the diet is automatically converted to RSS, thanks to the. rss extension in the 
URL. 



mywiki rss Feed 

mywiki news 
My Main Page 

11/23/2011 09:38 AM 

Hello tod ay I made my first wild application 


web2py also includes feedparser to read third-party feeds. 

Finally, let's add an XML-RPC handler that allows searching the wiki 
programmatically: 


service = Service! ) 







(aservice.xmlrpc 

def find_by(keyword) : 
"finds pages that 
return db(db.page 

contain keyword for XML- 
title. contains (keyword) 

RPC" 
select( ) .as 

-list() 

def 

caU(): 

"exposes all regis 

return service!) 

tered 

services. 

including XML 

RPC" 



Here, the handler action simply publishes (via XML-RPC), the functions 
specified in the list. In this case, f incLby. f incLby is not an action (because it 
takes an argument). It queries the database with . select () and then extracts 
the records as a list with . response and returns the list. 
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Here is an example of how to access the XML-RPC handler from an external 
Python program. 

1 >» import xmlrpclib 

2 >» server = xmlrpclib. ServerProxy( 

3 ' bttp: / 1127 .B .B .l:8BBB/myviiki/ default! call/xmlrpc') 

4 >» for item in server. find_by( 'wiki ') : 

5 print item [' created-on' ] , item[ ' title' ] 

The handler can be accessed from many other programming languages that 
understand XML-RPC, including C, C++, C# and Java. 


J.10.1 On date, datetime and time format 

There are three different representation for each of the field types date, 
datetime and time: 

• the database representation 

• the internal web2py prepresentation 

• the string representation in forms and tables 

The database representation is an internal issue and does not affect the 
code. Internally, at the web2py level, they are stored as datetime. date, 
datetime. datetime and datetime. time object respectively and they can be 
manipulated as such: 

1 for page in db(db. page) .select () : 

2 print page. title, page. day, page. month, page. year 

When dates are converted to strings in forms they are converted using the 
ISO representation 

, %Y-%m-%d %H:%M:%S 

yet this representation in internationalized and you can use the admin 
stranslation page to change the format to an alternate one. For example: 

, %m/%b/%Y %H:%M:%S 

Mind that by default English is not translated because webzpy assumes the 
applications is already written in English. If you want internationalization 
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to work for English you need to create the translation file (using admin) and 
you need declare that the application current language is something other than 
english,for example: 

T ,current_languages = ['null'] 


3.11 More on admin 


The administrative interface provides additional functionality that we briefly 
review here. 


3.11.1 site 

This page lists all installed applications. There are two forms at the bottom. 

The first of them allows creating a new application by specifying its name. 

The second form allows uploading an existing application from either a local 
file or a remote URL. When you upload an application, you need to specify 
a name for it. This can be its original name, but does not need to be. This 
allows installing multiple copies of the same application. You can try, for 
example, to upload the the Instant Press CMS created by Martin Mulone 
from: 

http://code.google.eom/p/instant-press/ 

Web2py files are packages as . w2p files. These ones are tar gzipped files. 
Web2py uses the .w2p extension instead of the .tgz extension to prevent the 
browser from unzipping on download. They can be uncompressed manually 
with tar zxvf [filename] although this is never necessary. 
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O Upload and install packed application 


Application 
name: 
Upload a 
package: 

Get from URL: 


| Instant Press 

( Choose File | (NorKj) 
OR 

|)ittp:ffweb2py_com/applia| 
O Overwrite Installed app 

[ install | 


Upon successful upload, web2py displays the MD5 checksum of the 
uploaded file. You can use it to verify that the file was not corrupted 
during upload. The InstantPress name will appear in the list of installed 
applications. 

Click on the InstantPress name on admin to get it up and running. 

go < : ' «^i"s. "■"" »»™— «"*■■» 




' 3W"?*WS-«P*T1«C 


rttaa ■ ..- ■ .' ' f- ■ ■ ■ 


You can read more about Instant Press at the following URL: 

http://code.google.eom/p/instant-press/ 


For each application the site page allows you to: 

• Uninstall the application. 

• Jump to the about page (read below). 
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• Jump to the edit page (read below). 

• Jump to the errors page (read below). 

• Clean up temporary files (sessions, errors, and cache. disk files). 

• Pack all. This returns a tar file containing a complete copy of the 
application. We suggest that you clean up temporary files before packing 
an application. 

• Compile the application. If there are no errors, this option will bytecode- 
compile all models, controllers and views. Because views can extend and 
include other views in a tree, before bytecode compilation, the view tree 
for every controller is collapsed into a single file. The net effect is that a 
bytecode-compiled application is faster, because there is no more parsing 
of templates or string substitutions occurring at runtime. 

• Pack compiled. This option is only present for bytecode-compiled 
applications. It allows packing the application without source code 
for distribution as closed source. Note that Python (as any other 
programming language) can technically be decompiled; therefore 
compilation does not provide complete protection of the source code. 
Nevertheless, decompilation can be difficult and can be illegal. 

• Remove compiled. It simply removes the byte-code compiled models, 
views and controllers from the application. If the application was 
packaged with source code or edited locally, there is no harm in removing 
the bytecode-compiled files, and the application will continue to work. If 
the application was installed form a packed compiled file, then this is not 
safe, because there is no source code to revert to, and the application will 
no longer work. 

All the functionality available from the webzpy admin site page is also 
accessible programmatically via the API defined in the module gluon/admin . py. 
Simply open a python shell and import this module. 
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3.11.2 about 


The about tab allows editing the description of the application and its license. 
These are written respectively in the ABOUT and LICENSE files in the 
application folder. 



Y^Y^YJ^ 


• ABOUT APPLICATION "WELCOME" . 
O About welcome 

HI 


Write something about this app. Developed with wet>2py. 


License for welcome 

Edit 








The wet>2py -welcome app is lies 

nsed under pu 


*-*«.*. 

— ■» — - 

'"•- 

ave their 

ivn third party 

You can modify this license wh 

„»..™».u 

.0 















You can use MARKMIN, or gluon.contrib.markdown.WIKI syntax for these files as 
described in ref. [29]. 


3.11.3 edit 

You have used the edit page already in this chapter. Here we want to point 
out a few more functionalities of the edit page. 

• If you click on any file name, you can see the contents of the file with 
syntax highlighting. 

• If you click on edit, you can edit the file via a web interface. 

• If you click on delete, you can delete the file (permanently). 

• If you click on test, web2py will run tests. Tests are written by the 
developer using Python doctests, and each function should have its own 
tests. 


• You can add language files, scan the app to discover all strings, and edit 
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string translations via the web interface. 

• If the static files are organized in folders and subfolders, the folder 
hierarchy can be toggled by clicking on a folder name. 

The image below shows the output of the test page for the welcome 
application. 



Tf Versioning Tf Logout Jf Help jU 


■ TESTING SLCOME" 


• TESTIf' J FY" ... DONE. 


• Function ccache [no doctests] 

• Function csv [no doctests] 

• Function download [no doctests] 

• Function eval_in_global_env [no doctests] 

• Function get_database [no doctests] 

• Function get_databases [no doctests] 

• Function get_query [no doctests] 

• Function get_lable [no doctests] 


The image below show the languages tab for the welcome application. 


OVERVIEW 121 



L^ http://127.0.0.1:8000/adm 

n/default/design/welcome % Cf ^Back @ | 

m - ■,.■*-■*:■.. . 

^M Site U Edit W About W Errors W Verslon.infl W Loqout M Help W 

• EDIT APPLICATION 

"WELCOME" 

^^^fflEil models controllers views Languages static modules pluglns 

s>\ \m 

O Models « 


O Controllers© 


Views ki 


Languages© 


B r. —, 


UdU 1^ es-es.py 


jafl ■ i.-c..p, 


SB . Ir-Jr bv 


LtdtJ ^ M-hl.py 


SB -. I,»-I,» p, 


SB -. hu.py 


SB -, ii-,i P , 


SB Ir. ii p, 


SB Ir. pi.gi.pv 


SB I'm pi.py 


SIB TC Pt-br.Pv 


SB ■ ptpipy 


LEOU - ptpv 


SB K ™~p/ 


LfiH^ 1^, sK-sK.py 


1 faMJ |^, zh-tw.py 



create file with "filename: 

l|C«.| 

(something like "It-ff") 




The image below shows how to edit a language file, in this case the "it" 
(Italian) language for the welcome application. 



• EDITING LANGUAGE FILE "WELCOME/LANGUAGES/PT-BR.PY" 
•OR IG INAL/TRANS LAJION 

. update or delete the results of a JOIN 


■;&.?■ 

Is an optional ex 

session Ilk 

"field- 

='newva 



e uma expressa 



rtipol-'novova.lor" 

.'CCS na 

a pode atualizar- 



dos de 

m JOIN 


%s imr 

as apagadas 

l| delete] 

*,™ 

updated 


%5 111 

as atuallzadas 

dsk*l 

%Y-%rr 

-%d 


%d-%n 

-%Y 

| delete | 




%Y-%rr 



%d-%n 

-%Y %H:%M:%3 

|| delete] 

About 

AQtJUt 




shell 


122 WEB2PY FULL-STACK WEB FRAMEWORK, 4TH EDITION 


If you click on the "shell" link under the controllers tab in edit, web2py will 
open a web based Python shell and will execute the models for the current 
application. This allows you to interactively talk to your application. 



~^l Edit If About V Errors V Versioning V Logout V Helf> W 


' ■ ■" 


crontab 

Also under the controllers tab in edit there is a "crontab" link. By clicking 
on this link you will be able to edit the web2py crontab file. This follows 
the same syntax as the unix crontab but does not rely on unix. In fact, it 
only requires web2py, and it works on Windows. It allows you to register 
actions that need to be executed in background at scheduled times. For more 
information about this, see the next chapter. 


3.11.4 errors 


When programming web2py, you will inevitably make mistakes and 
introduce bugs. web2py helps in two ways: 1) it allows you to create tests 
for every function that can be run in the browser from the edit page; and 2) 
when an error manifests itself, a ticket is issued to the visitor and the error is 
logged. 

Intentionally introduce an error in the images application as shown below: 
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When you access the index action, you get the following ticket: 


C^ I^J h ttp://lZ7.0.0.1:8000/images 

Internal error 


_♦ e *-Back & 


Ticket issued: imaaes/127. 0.0. 1.2011-11-23. 12-01-55. f5154d95-14'6d-4a38-8545-87e0daf206bb 


Only the administrator can access the ticket: 


L^A, I >iitp://1 27.0.0.1 :8000/admin/clefaLltAi[ket/i mages/ 127.0. 0.1. 201 1-1 1-2j.12-01-55.f 51 5 4d 9^ . - Bart 

Ljhttp://127.0.0... X ^ ticket images/... ■ 

Errors jf Verslonlng V Logout V Help ■ 


• ERROR 1 "IMAGES" 


1Z7.0.0.1.2011-1 1-23. 12-0 1-55. 15 154rJ.95-1 4 Sd-4a38-8 545-87 e»aar20&Db 

<type 'exceptions. ZeroDivisionError'> integer division or modulo by zi 


web2py™ (1. 99. 3. datetlme.<faletlme(2011. 11. 21. 23. 29. 32). '^ev') 
Python Python 2.7.2+: /usr.'Bln/python 


TRACEBACK 


sclf._cfll"ter - Ian 


ERROR SNAPSHOT 6 






<type 'exceptic 


The ticket shows the traceback, and the content of the file that caused the 
problem, and the complete state of system (variables, request, session, etc.) 
If the error occurs in a view, web2py shows the view converted from HTML 
into Python code. This allows to easily identify the logical structure of the 
file. 
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By default tickets are stored on filesystem and group by traceback. The 
administrative interface provides an aggregate views (type of traceback and 
number of occurrence) and a detailed view (all tickets are listed by ticket id). 
The administrator can switch between the two views. 

Notice that everywhere admin shows syntax-highlighted code (for example, 
in error reports, web2py keywords are shown in orange). If you click on 
a web2py keyword, you are redirected to a documentation page about the 
keyword. 

If you fix the divide-by-zero bug in the index action and introduce one in the 
index view: 


1 

{{extend ' layout. html ' }} 


3 

<hl>Current Images</hl> 


4 

<ul> 


5 

{{for image in images:}} 


6 

{{1/0}} 


7 

{{=LI(A( image. title, _href=URL( "show" 

args=image.id) ) )}} 

8 

{{pass}} 


9 

</ul> 



you get the following ticket: 


i-jhttp://127.0.0.... «|liI ticket images/... « 


L3, http://T 27.0. 0.1:8000/admin/defaLlt/ticket/images/1 27.0.0.1. 20 1 1-1 1-23.1 2-03-50.1 a90aa: ...■ .-Back 


/book./web2p>./gl_Ljon/r 


ERROR SNAPSHOT « 


(type exceptii 


Frames 
Flle/hQme/bTiice/proJects/book/web2py/gluon/restrlcted.py li 


File i home.'briice/projects/baok/weblpy/ application si ImagesJi'iewsldefa ulUindex.html In 'module* at line 78 code arguments 


Function argument list 


class=" , escape=Fals« 
, e5caoe=FaTseI 


Note that web2py has converted the view from HTML into a Python file, and 
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the error described in the ticket refers to the generated Python code and NOT 
to the original view file: 


!_{$ http://1 27.0.0. 1:8O00/admirVdefault/tkkeS/image5/127.Q.Q.1.2Q1 1-1 1-23.1 2-03-5Q.1a9Oaa3e4fbl* ^ (jrBack \JJ 


http://127.0.O.... X ticket images/... X 


!<middlt 
■(' coluu 




div class--', escape=F*Ase 
, e 5 cape=Fals*) 


sponse.writel 1 ,n 


E a]!»!htnl clas5="ie ieB le-lteS le-lteB rm-js" lar 
E 5]><h-tml class="ieS i_=-lte9 nn-js n lang='", escap 


This may seem confusing at first, but in practice it makes debugging easier, 
because the Python indentation highlights the logical structure of the code 
that you embedded in the views. 

The code is shown at the bottom of the same page. 

All tickets are listed under admin in the errors page for each application: 
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^ (tu hLtp://127.0.0.1:8000/admin/default/erfors/images %j ^J ^ Back cj 

y http://l 27.0.0.... ■ errors images ■ 

Site M Edit if About If Errors if Versioning if Logout if Help 

• ERROR LOGS FOR IMAGES" 

check all unc heck all delete all checked 

O Click row to expand traceback 

source : filesyslem switch to : db lists by ticket 


1 Index. ntml ZoBDMelonErnr: Integer division or modulo t>y zero + details 

"I default.py ZeroDivis ion Error: integer division or modulo by zero + details 



3.11.5 Mercurial 

If you are running from source and you have the Mercurial version control 
libraries installed: 

easy_install mercurial 


then the administrative interface shows one more menu item called 
"mercurial". It automatically creates a local Mercurial repository for the 
application. Pressing the "commit" button in the page will commit the current 
application. Mercurial creates and stores information about changes you 
make in your code into a hidden folder ".hg" in your app subfolder. Every 
app has its own "hg" folder and its own "hgignore" file (tells Mercurial which 
files to ignore). 

The Mercurial web interface does allow you to browse previous commit and 
diff files but we do recommend you use Mercurial directly from the shell or 
one of the may GUI-based Mercurial clients since they are more powerful. For 
example they will allow you sync your app with a remote source repository: 
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C^ [lu http://l27.0.0.1:8DDD/admir/mercLrial/commit/images 4 | ^ ^j» Back ^ (§) 

lj http://1 27.0.0. ... II L_,commil; images ** 

Sits V Edit M About M Errors V Versioning V Logout V Help \m 

IMERCURIAL VERSION CONTROL SYSTEM INTERFACE 
FOR APPLICATION "IMAGES" 
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You can read more about Mercurial here: 

http: //mercurial. selenic. com/ 

3.11.6 Admin wizard (experimental) 



The admin interface includes a Wizard that can help you create a new 
applications. You can access the wizard from the "sites" page as shown in 
the image below. 
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The wizard will guide you through a series of steps involved in creating a 
new application: 

• Chose a name for the application 
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• Configure the application and choose required plugins 

• Build required models (it will create CRUD pages for each model) 

• Allow you to edit the views of those pages using MARKMIN syntax 
The image below shows the second step of the process. 
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You can see a dropdown to select a layout plugin (from web2py.com/layouts), 
a multiple choice dropdown to check other plugins (from web2py.com/plugins) 
and a "login config" field where to put the Janrain "domain:key". 

The other steps are pretty much self-explanatory. 

The Wizard works well for what it does but it is considered an experimental 
feature for two reasons: 

• Applications created with the wizard and edited manually, cannot later be 
modified by the wizard. 

• The interface of the wizard will change over time to include support for 
more features and easier visual development. 

In any case the wizard is a handy tool for fast prototyping and it can be used 
to bootstrap a new application with an alternate layout and optional plugins. 
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3.11.7 Configuring admin 

Normally there is no need to perform any configuration of admin but a few 
customizations are possible. After you login into admin you can edit the 
admin configuration file via the URL: 
http : //127 . 0.0 . 1 : 8000/admin/def ault/edit/admin/models/0 . py 

Notice that admin can be used to edit itself. In fact admin is an app as any 
other one. 

The file "o.py" is very much self documented and if you are opening 
probably you already know what you are looking for. Anyway there a few 
customizations that are more important than others: 
GAE_APPCFG = os . path .abspath(os . path . join ( ' /us r/ local/ bin/ appcfg.py' ) ) 

This should point to the location of the "appcfg.py" file that comes with the 
Google App Engine SDK. If you have the SDK you may want to change this 
config parameters to the correct value. It will allow you to deploy to GAE 
from the admin interface. 

You can also set web2py admin in demo mode: 

DEM0_M0DE = True 
FILTER_APPS = ['welcome'] 

And only the apps listed in filter apps will be accessible and they will be only 
accessible in read-only mode. 

If you are a teacher and want to expose the administrative interface to 
students so that students can share one administrative interface for their 
projects (think of a virtual lab), can do it by setting: 

MULTI_USER_MODE = True 

In this way students will be required to login and will only be able to access 
their own apps via admin. You, as first user/teacher, will be able to access 
them all. 

Mind that this mechanism still assumes all users are trusted. All the apps 
created under admin run under the same credentials on the same filesystem. 
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It is possible for an app created by a student to access the data and the source 
of an app created by another student. 


3.22 More on appadmin 

appadmin is not intended to be exposed to the public. It is designed to help 
you by providing an easy access to the database. It consists of only two files: 
a controller "appadmin. py" and a view "appadmin.html" which are used by 
all actions in the controller. 

The appadmin controller is relatively small and readable; it provides an 
example of designing a database interface. 

appadmin shows which databases are available and which tables exist in 
each database. You can insert records and list all records for each table 
individually, appadmin paginates output 100 records at a time. 

Once a set of records is selected, the header of the pages changes, allowing 
you to update or delete the selected records. 

To update the records, enter an SQL assignment in the Query string field: 

title = 'test' 

where string values must be enclosed in single quotes. Multiple fields can be 
separated by commas. 

To delete a record, click the corresponding checkbox to confirm that you are 
sure. 

appadmin can also perform joins if the SQL FILTER contains a SQL condition 
that involves two or more tables. For example, try: 

db. image. id == db. comment . image_id 

web2py passes this along to the DAL, and it understands that the query links 
two tables; hence, both tables are selected with an INNER JOIN. Here is the 
output: 
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If you click on the number of an id field, you get an edit page for the record 
with the corresponding id. 

If you click on the number of a reference field, you get an edit page for the 
referenced record. 

You cannot update or delete rows selected by a join, because they involve 
records from multiple tables and this would be ambiguous. 

In addition to its database administration capabilities, app admin also 
enables you to view details about the contents of the application's cache 
(at /yourapp/appadmin/ccache) as well as the contents of the current request, 
response, and session objects (at /yourapp/appadmin/state). 

appadmin replaces response. menu with its own menu, which provides links 
to the application's edit page in admin, the db (database administration) 
page, the state page, and the cache page. If your application's layout 
does not generate a menu using response. menu, then you will not see the 
appadmin menu. In that case, you can modify the appadmin.html file and 
add {{=MENU(response.menu)}} to display the menu. 


4 

The core 


4.1 Command line options 

It is possible to skip the GUI and start web2py directly from the command 

line by typing something like: 

python web2py.py -a 'your password' -i 127.0.0.1 -p 8000 

When web2py starts, it creates a file called "parameters_8ooopy" where it 
stores the hashed password. If you use "<ask>" as the password, web2py 
prompts you for it. 

For additional security, you can start web2py with: 

python web2py.py -a ' <recycle>' -i 127.0.0.1 -p 8000 

In this case web2py reuses the previously stored hashed password. If no 
password is provided, or if the "parameters_8ooopy" file is deleted, the web- 
based administrative interface is disabled. 

On some Unix /Linux systems, if the password is 
<pam_user: some_user> 

web2py uses the PAM password of the Operating System account of 
some_user to authenticate the administrator, unless blocked by the PAM 
configuration. 
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webzpy normally runs with CPython (the C implementation of the Python 
interpreter created by Guido van Rossum), but it can also run with Jython 
(the Java implementation of the interpreter). The latter possibility allows 
the use of webipy in the context of a J2EE infrastructure. To use Jython, 
simply replace "python webipy.py..." with "jython webzpy.py" . Details about 
installing Jython, zxJDBC modules required to access the databases can be 
found in Chapter 14. 

The "web2py.py" script can take many command-line arguments specifying 
the maximum number of threads, enabling of SSL, etc. For a complete list 
type: 

1 >» python web2py.py -h 

2 Usage: python web2py.py 
3 

4 web2py Web Framework startup script. ATTENTION: unless a password 

5 is specified (-a ' passwd' ) , web2py will attempt to run a GUI. 

6 In this case command line options are ignored. 
7 

8 Options: 

9 --version show program's version number and exit 
in -h, --help show this help message and exit 

u -i IP, --ip=IP ip address of the server (127. Q.Q.I) 

.2 -p PORT, --port=P0RT port of server (8QQQ) 

13 -a PASSWORD, --password=PASSWORD 

14 password to be used for administration (use -a 

15 "<recycle>" to reuse the last password) ) 
,6 -c SSL-CERTIFICATE, - -ssl-Certificate=SSL-CERTIFICATE 

17 file that contains ssl certificate 
„s -k SSL ^PRIVATE. KEY , - -ssl-private_key=SSL-PRIVATE-KEY 

19 file that contains ssl private key 

-d PID- FILENAME, - -pid.filename=PID.FILENAME 

21 file to store the pid of the server 

-I LOG.FILENAME, - -log.filename=LOG-FILENAME 

23 file to log connections 

24 -n NUNTHREADS, - -numthreads=NUMTHREADS 

25 number of threads (deprecated) 

26 --minthreads=MINTHREADS 

27 minimum number of server threads 

28 - -maxthreads=MAXTHREADS 

29 maximum number of server threads 

30 -s SERVER-NAME, - -server-name=SERVER.NAME 

31 server name for the web server 

32 -q REQUEST-QUEUE-SIZE, - - request-queuesize=REQUEST-QUEUESIZE 

33 max number of queued requests when server unavailable 
,4 -0 TIMEOUT, --timeout=TIMEOUT 
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timeout for individual request (10 seconds) 

-z SHUTDOWN-TIMEOUT, 

--shutdown-timeout=SHUTDOWN-TIMEOUT 


timeout on shutdown of server (5 seconds) 

-f FOLDER, --folder=FOLDER 


folder from which to run web2py 

-v, --verbose 

increase --test verbosity 

-0, --quiet 

disable all output 

-D DEBUGLEVEL, - -debug=DEBUGLEVEL 


set debug output level (0-100, means all, 100 means 


none; default is 30) 

-S APPNAME, --shell=APPNAME 


run web2py in interactive shell or IPython (if 


installed) with specified appname (if app does not 


exist it will be created). APPNAME like a/c/f (c,f 


optional) 

-B, --bpython 

run web2py in interactive shell or bpython (if 


installed) with specified appname (if app does not 


exist it will be created). Use combined with --shell 

-P, --plain 

only use plain python shell; should be used with 


--shell option 

-M, - -import-models 

auto import model files; default is False; should be 


used with --shell option 

-R PYTHON-FILE, - - run=PYTHON-FILE 


run PYTHON-FILE in web2py environment; should be used 


with --shell option 

-K SCHEDULER, - - schedule r=SCHEDULER 


run scheduled tasks for the specified apps 


-K appl, app2, app3 requires a scheduler defined in the 


models of the respective apps 

-T TEST_PATH, --test-- 

--TEST-PATH 


run doctests in web2py environment; TEST-PATH like 


a/c/f (c,f optional) 

-W WINSERVICE, - -winservice=WINSERVICE 


-W install\start\stop as Windows service 

-C, --cron 

trigger a cron run manually; usually invoked from a 


system cron tab 

- -softcron 

triggers the use of softcron 

-N, --no-cron 

do not start cron automatically 

-J, --cron job 

identify cron- initiated command 

-L CONFIG, --config=CONFIG 


config file 

-F PROFILER-FILENAME, 

- -profiler=PROFILER_FILENAME 


profiler filename 

-t, --taskbar 

use web2py gui and run in taskbar (system tray) 

- -nogui 

text-only, no GUI 

-A ARGS, --args=ARGS 

should be followed by a list of arguments to be passed 


to script, to be used with -S, -A must be the last 


option 

- -no-banner 

Do not print header banner 
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84 

- -interfaces=INTERFACES 


85 

listen on multiple addresses: 


86 

"ip:port:cert:key;ip2:port2:cert2:key2; . . 

. " (: cert: key 

87 

optional; no spaces) 



Lower-case options are used to configure the web server. The - L option tells 
web2py to read configuration options from a file, -W installs web2py as a 
windows service, while -S, -P and -M options start an interactive Python 
shell. The -T option finds and runs controller doctests in a web2py execution 
environment. For example, the following example runs doctests from all 
controllers in the "welcome" application: 

1 python web2py.py -vT welcome 

if you run web2py as Windows Service, -W, it is not convenient to pass the 
configuration using command line arguments. For this reason, in the web2py 
folder there is a sample "options_std.py" configuration file for the internal 
web server: 

1 import socket 

2 import os 
3 

4 ip = '0.0.0.0' 

5 port = 80 

6 interfaces=[( '0.0.0.0' ,80)] 

- #interfaces.append( (' 0.0.0.0' ,443, ' ssl_private_key.pem' , ' ssl_certificate.pem' ) ) 

s password = '<recycle>' # ## <recycle> means use the previous password 

i) picLfilename = ' httpserver.pid' 

io log-filename = ' httpserver. log' 

u profiler. filename = None 

iz minthreads = None 

i 3 maxthreads = None 

i 4 server_name = socket .gethostname( ) 

15 request_queue_size = 5 

16 timeout = 30 

i 7 shutdown_timeout = 5 

18 folder = os.getcwd() 

h, extcron = None 

20 nocron = None 

This file contains the web2py defaults. If you edit this file, you need to 
import it explicitly with the -L command-line option. It only works if you 
run web2py as a Windows Service. 
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4.2 Workflow 


The web2py workflow is the following: 

• An HTTP requests arrives to the web server (the built-in Rocket server or 
a different server connected to web2py via WSGI or another adapter). The 
web server handles each request in its own thread, in parallel. 

• The HTTP request header is parsed and passed to the dispatcher 
(explained later in this chapter). 

• The dispatcher decides which of the installed application will handle the 
request and maps the PATH_INFO in the URL into a function call. Each 
URL corresponds to one function call. 

• Requests for files in the static folder are handled directly, and large files 
are automatically streamed to the client. 

• Requests for anything but a static file are mapped into an action (i.e. a 
function in a controller file, in the requested application). 

• Before calling the action, a few things happen: if the request header 
contains a session cookie for the app, the session object is retrieved; if 
not, a session id is created (but the session file is not saved until later); an 
execution environment for the request is created; models are executed in 
this environment. 

• Finally the controller action is executed in the pre-built environment. 

• If the action returns a string, this is returned to the client (or if the action 
returns a web2py HTML helper object, it is serialized and returned to the 
client). 

• If the action returns an iterable, this is used to loop and stream the data to 
the client. 

• If the action returns a dictionary, web2py tries to locate a view to 
render the dictionary. The view must have the same name as the action 
(unless specified otherwise) and the same extension as the requested page 
(defaults to. html); on failure, web2py may pick up a generic view (if 
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available and if enabled). The view sees every variable defined in the 
models as well as those in the dictionary returned by the action, but does 
not see global variables defined in the controller. 

• The entire user code is executed in a single transaction unless specified 
otherwise. 

• If the user code succeeds, the transaction is committed. 

• If the user code fails, the traceback is stored in a ticket, and a ticket ID is 
issued to the client. Only the system administrator can search and read 
the tracebacks in tickets. 

There are some caveats to keep in mind: 

• Models in the same folder /sub folder are executed in alphabetical order. 

• Any variable defined in a model will be visible to other models following 
alphabetically, to the controllers, and to the views. 

• Models in subfolders are executed conditionally. For example, if the user 
has requested "/a/c/f" where "a" is the application, "c" is the controller, 
and "f" is the function (action), then the following models are executed: 

applications/a/models/* . py 
applications/a/models/c/* . py 
applications/a/models/c/f /* . py 

• The requested controller is executed and the requested function is called. 
This means all top-level code in the controller is also executed at every 
request for that controller. 

• The view is only called if the action returns a dictionary. 

• If a view is not found, web2py tries to use a generic view. By default, 
generic views are disabled, although the 'welcome' app includes a line in 
/ models /db.py to enable them on localhost only. They can be enabled 
per extension type and per action (using response. generic_patterns). In 
general, generic views are a development tool and typically should not be 
used in production. If you want some actions to use a generic view, list 
those actions in response. generic_patterns (discussed in more detail in the 
chapter on Services). 
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The possible behaviors of an action are the following: 

Return a string 

def index(): return 'data' 

Return a dictionary for a view: 

def index(): return diet ( key= ' value' ) 

Return all local variables: 

def index(): return locals() 

Redirect the user to another page: 

def index(): redirect ( URL ( 'other-action ') ) 

Return an HTTP page other than "200 OK": 

def index(): raise HTTP(404) 

Return a helper (for example, a FORM): 

def index(): return FORM ( INPUT (_name=' test ') ) 

(this is mostly used for Ajax callbacks and components, see chapter 12) 

When an action returns a dictionary, it may contain code generated by 
helpers, including forms based on database tables or forms from a factory, 
for example: 

def index(): return dict(form=SQLFORH.factory(Field( 'name ')). process( ) ) 

(all forms generated by web2py use postbacks, see chapter 3) 

4.3 Dispatching 

web2py maps a URL of the form: 

http://127.0.0.1:8000/a/c/f .html 

to the function f ( ) in controller "c.py" in application "a". If f is not present, 
web2py defaults to the index controller function. If c is not present, web2py 
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defaults to the "default.py" controller, and if a is not present, web2py defaults 
to the init application. If there is no init application, web2py tries to run the 
welcome application. This is shown schematically in the image below: 

(The names of the default application, controller and function can be 
overridden in routes. py; see Default Application, Controller and Function below. 
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By default, any new request also creates a new session. In addition, a session 
cookie is returned to the client browser to keep track of the session. 

The extension . html is optional; . html is assumed as default. The extension 
determines the extension of the view that renders the output of the controller 
function f ( ) . It allows the same content to be served in multiple formats 
(html, xml, json, rss, etc.). 

Functions that take arguments or start with a double underscore are not 
publicly exposed and can only be called by other functions. 

There is an exception made for URLs of the form: 
http ://127 .9.8 . 1: 8000/a/static/f ilename 


There is no controller called "static". web2py interprets this as a request for 
the file called "filename" in the subfolder "static" of the application "a". 

When static files are downloaded, web2py does not create a session, nor 
does it issue a cookie or execute the models. web2py always streams static 
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files in chunks of 1MB, and sends PARTIAL CONTENT when the client 
sends a RANGE request for a subset of the file. web2py also supports the 
IF_MODIFIED_SINCE protocol, and does not send the file if it is already 
stored in the browser's cache and if the file has not changed since that 
version. 

When linking to an audio or video file in the static folder, if you want to 
force the browser to download the file instead of streaming the audio /video 
via a media player, add ?attachment to the URL. This tells web2py to set 
the Content -Disposition header of the HTTP response to "attachment". For 
example: 
1 <a href ="/app/ stat ic/my_audio_f He. mp3?attachinent ">Download</a> 

When the above link is clicked, the browser will prompt the user to download 
the MP3 file rather than immediately streaming the audio. (As discussed 
below, you can also set HTTP response headers directly by assigning a diet of 
header names and their values to response, headers.) 

web2py maps GET/POST requests of the form: 

1 http : //127 . . . 1 : 8000/a/c/f . html/x/y/z?p=l&q=2 

to function f in controller "c.py" in application a, and it stores the URL 
parameters in the request variable as follows: 
■ request. args = ['x', 'y 1 , 'z'] 

and: 

] request. vars = {'p':l, l q':2} 

and: 

1 request .application = 'a' 

2 request .controller = 'c' 

3 request .function = 'f 

In the above example, both request .args[i] and request . args(i) can be used 
to retrieve the i-th element of the request . a rgs, but while the former raises an 
exception if the list does not have such an index, the latter returns None in 
this case. 
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request .url 


stores the full URL of the current request (not including GET variables), 
request. a j ax 

defaults False but it is True if web2py determines that the action was called 
by an Ajax request. 

If the request is an Ajax request and it is initiated by a web2py component, 
the name of the component can be found in: 
request. cid 

Components are discussed in more detail in Chapter 12. 

If the HTTP request is a GET, then request . env . request_method is set to "GET"; 
if it is a POST, request. env. request_method is set to "POST". URL query 
variables are stored in the request. vars Storage dictionary; they are also 
stored in request .get_vars (following a GET request) or request . post_vars 
(following a POST request). web2py stores WSGI and web2py environment 
variables in request. env, for example: 
request .env. path_info = 'a/c/f 

and HTTP headers into environment variables, for example: 

request. env. http.host = ' 127.0.0. 1:8000' 

Notice that webipy validates all URLs to prevent directory traversal attacks. 

URLs are only allowed to contain alphanumeric characters, underscores, and 
slashes; the args may contain non-consecutive dots. Spaces are replaced by 
underscores before validation. If the URL syntax is invalid, web2py returns 
an HTTP 400 error message [47, 48]. 

If the URL corresponds to a request for a static file, web2py simply reads and 
returns (streams) the requested file. 

If the URL does not request a static file, web2py processes the request in the 
following order: 

• Parses cookies. 
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• Creates an environment in which to execute the function. 

• Initializes request, response, cache. 

• Opens the existing session or creates a new one. 

• Executes the models belonging to the requested application. 

• Executes the requested controller action function. 

• If the function returns a dictionary, executes the associated view. 

• On success, commits all open transactions. 

• Saves the session. 

• Returns an HTTP response. 

Notice that the controller and the view are executed in different copies of 
the same environment; therefore, the view does not see the controller, but 
it sees the models and it sees the variables returned by the controller action 
function. 

If an exception (other than HTTP) is raised, web2py does the following: 

• Stores the traceback in an error file and assigns a ticket number to it. 

• Rolls back all open transactions. 

• Returns an error page reporting the ticket number. 

If the exception is an HTTP exception, this is assumed to be the intended 
behavior (for example, an HTTP redirect), and all open database transactions 
are committed. The behavior after that is specified by the HTTP exception 
itself. The HTTP exception class is not a standard Python exception; it is 
defined by web2py 


4.4 Libraries 

The web2py libraries are exposed to the user applications as global objects. 
For example (request, response, session, cache), classes (helpers, validators, 
DAL API), and functions (T and redirect). 
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These objects are defined in the following core files: 

web2py.py 

gluon/ init py gluon/highlight .py gluon/ restricted. py gluon/streamer. py 

gluon/admin. py gluon/html.py gluon/ rewrite. py gluon/template.py 

gluon/cache. py gluon/http.py gluon/rocket .py gluon/storage. py 

gluon/cfs.py gluon/import_all.py gluon/sanitizer. py gluon/tools .py 

gluon/compileapp.py gluon/languages.py gluon/serializers . py gluon/utils .py 

gluon/contenttype.py gluon/main.py gluon/settings. py gluon/validators.py 

gluon/dal.py gluon/myregex.py gluon/shell.py gluon/widget . py 

gluon/decoder.py gluon/newcron.py gluon/sql.py gluon/winservice.py 

gluon/fileutils .py gluon/portalocker.py gluon/sqlhtml. py gluon/xmlrpc. py 

gluon/globals . py gluon/reserved_sql_keywords . py 

The tar gzipped scaffolding app that ship with web2py is 
welcome. w2p 

It is created upon installation and overwritten on upgrade. 

The first time you start webipy, two new folders are created: deposit and 
applications. The "welcome" app is zipped into a " welcome. wzp" file to be 
used as a scaffolding app. The first time you start webipy, two new folders 
are created: deposit and applications. The "welcome" app is zipped into a 
"welcome.wip" file to be used as scaffolding app. The deposit folder is used as 
temporary storage for installing and uninstalling applications. 

web2py unit-tests are in 

gluon/tests/ 

There are handlers for connecting with various web servers: 


cgihandler. py 


# discouraged 

gaehandler. py 


# for Google App Engine 

fcgihandler. py 


# for FastCGI 

wsgihandler. py 


# for WSGI 

isapiwsgihandler 

■ py 

# for IIS 

modpythonhandler 

■ py 

# deprecated 


("fcgihandler" calls "gluon/contrib/gateways/fcgi.py" developed by Allan 
Saddi) and 


anyserver.py 
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which is a script to interface with many different web servers, described in 
Chapter 13. 

There are three example files: 

1 options_std. py 

2 routes. example. py 

3 router. example. py 

The former is an optional configuration file that can be passed to web2py.py 
with the -L option. The second is an example of a URL mapping file. It is 
loaded automatically when renamed "routes. py". The third is an alternative 
syntax for URL mapping, and can also be renamed (or copied to) "routes. py". 

The files 

■ app.yaml 

2 index. yaml 

3 queue. yaml 

are configuration files used for deployment on the Google App Engine. You 
can read more about them in the Deployment Recipes chapter and on the 
Google Documentation pages. 

There are also additional libraries, usually developed by a third party: 

feedparser [28] by Mark Pilgrim for reading RSS and Atom feeds: 

1 gluon/contrib/ init py 

2 gluon/contrib/feedparser. py 

markdown2 [29] by Trent Mick for wiki markup: 

1 gluon/cont rib/ma rkdown/ init py 

2 gluon/cont rib/ma rkdown/ma rkdown2 . py 

markmin markup: 

] gluon/cont rib/ma rkmin.py 

pyfpdf created my Mariano Reingart for generating PDF documents: 

■ gluon/contrib/pyfpdf 

This is not documented in this book but it is hosted and documented here: 
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] http://code.google.eom/p/pyfpdf/ 

pysimplesoap is a lightweight SOAP server implementation created by 
Mariano Reingart: 

] gluon/contrib/pysimplesoap/ 

simplejsonrpc is a lightweight JSON-RPC client also created by Mariano 
Reingart: 

1 gluon/contrib/simplejsonrpc. py 

memcache [30] Python API by Evan Martin: 

1 gluon/contrib/memcache/ init py 

2 gluon/contrib/memcache/memcache.py 

redis_cache is a module to store cache in the redis database: 

] gluon/contrib/redis_cache.py 

gql, a port of the DAL to the Google App Engine: 

1 gluon/contrib/gql.py 

memdb, a port of the DAL on top of memcache: 

] gluon/contrib/memdb.py 

gae_memcache is an API to use memcache on the Google App Engine: 

1 gluon/contrib/gae^memcache. py 

pyrtf [26] for generating Rich Text Format (RTF) documents, developed by 
Simon Cusack and revised by Grant Edwards: 

1 gluon/contrib/pyrtf 

2 gluon/contrib/pyrtf/ init py 

, gluon/cont rib/ pyrtf /Const ant s.py 

4 gluon/contrib/pyrtf/Elements . py 

5 gluon/cont rib/ pyrtf/P rope rtySet s.py 

6 gluon/contrib/pyrtf/README 

7 gluon/contrib/pyrtf/Renderer. py 

8 gluon/contrib/pyrtf/Styles . py 

PyRSSzGen [27] developed by Dalke Scientific Software, to generate RSS 
feeds: 
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] gluon/contrib/rss2.py 

simplejson [25] by Bob Ippolito, the standard library for parsing and writing 
JSON objects: 

1 gluon/cont rib/ simple j son/ init py 

z gluon/cont rib/ simple j son/decode r.py 

3 gluon/cont rib/ simple j son/encode r.py 

4 gluon/cont rib/ simple j son/j son filter. py 

5 gluon/cont rib/ simple json/s can ne r.py 

Google Wallet [96] provides "pay now" buttons which link Google as 
payment processor: 

1 gluon/contrib/google_wallet . py 

Stripe.com [98] provides a simple API for accepting credit card payments: 
1 gluon/contrib/stripe.py 

AuthorizeNet [99] provides API to accept credit card payments via 
Authorize.net network 

1 gluon/contrib/AuthorizeNet . py 

Dowcommerce [100] is yet another credit cart processing API: 

] gluon/contrib/DowCommerce.py 

PAM [75] authentication API created by Chris AtLee: 

1 gluon/contrib/pam.py 

A Bayesian classifier to populate the database with dummy data for testing 
purposes: 
1 gluon/contrib/populate.py 

A file that allows interaction with the taskbar in windows, when web2py is 
running as a service: 

1 gluon/contrib/taskbar_widget . py 

Optional login_methods and login_forms to be used for authentication: 
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gluon/contrib/login_methods/ init py 

gluon/contrib/login_methods/basic_auth.py 
gluon/cont rib/login_methods/cas_auth.py 
gluon/contrib/login_methods/dropbox_account . py 
gluon/cont rib/login_methods/email_auth . py 
gluon/contrib/login_methods/ext ended- login_ form.py 
gluon/cont rib/login_methods/gae_google_account .py 
gluon/cont rib/login_methods/ldap_auth.py 
gluon/contrib/login_methods/linkedin_account . py 
gluon/cont rib/login_methods/loginza.py 
gluon/contrib/login_methods/oauthl0a_account . py 
gluon/contrib/login_methods/oauth20_account . py 
gluon/cont rib/login_methods/openid_auth.py 
gluon/cont rib/login_methods/pam„auth . py 
gluon/cont rib/login_met hods/ rpx_account. py 
gluon/cont rib/login_methods/x509_auth.py 

web2py also contains a folder with useful scripts including 


sc ripts/set up -web2py- fedora. 

sh 


scripts/set up -web2py-ubuntu. 

sh 


sc ripts/set up- web2py-nginx- uwsgi • 

-ubuntu.sh 

script s/update-web2py.sh 



sc ripts/make_min_web2py . py 



scripts/sessions2trash.py 



sc ripts/sync_languages . py 



script s/tickets2db.py 



sc ripts/tickets2email . py 



sc ripts/ext ract_mysql_models 

;.py 


sc ripts/ext ract_pgsql_models 

;.py 


scripts/access .wsgi 



scripts/cpdb.py 




The first three are particularly useful because they attempt a complete 
installation and setup of a web2py production environment from scratch. 
Some of these are discussed in Chapter 14, but all of them contain a 
documentation string inside that explains their purpose and usage. 

Finally web2py includes these files required to build the binary distributions. 

1 Makefile 

2 setup_exe.py 

3 setup_app.py 
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These are setup scripts for py2exe and py2app, respectively, and they are 
only required to build the binary distributions of \veb2py YOU SHOULD 
NEVER NEED TO RUN THEM. 

In summary, web2py libraries provide the following functionality: 

• Map URLs into function calls. 

• Handle passing and returning parameters via HTTR 

• Perform validation of those parameters. 

• Protect the applications from most security issues. 

• Handle data persistence (database, session, cache, cookies). 

• Perform string translations for various supported languages. 

• Generate HTML programmatically (e.g. from database tables). 

• Generate SQL via a Database Abstraction Layer (DAL). 

• Generate Rich Text Format (RTF) output. 

• Generate Comma-Separated Value (CSV) output from database tables. 

• Generate Really Simple Syndication (RSS) feeds. 

• Generate JavaScript Object Notation (JSON) serialization strings for Ajax. 

• Translate wiki markup (Markdown) to HTML. 

• Expose XML-RPC web services. 

• Upload and download large files via streaming. 

web2py applications contain additional files, particularly third-party 
JavaScript libraries, such as jQuery, calendar /datepicker, Edit Area and 
nicEdit. Their authors are acknowledged in the files themselves. 


4.5 Applications 

Applications developed in web2py are composed of the following parts: 
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models describe a representation of the data as database tables and 
relations between tables. 

controllers describe the application logic and workflow. 

views describe how data should be presented to the user using HTML and 
JavaScript. 

languages describe how to translate strings in the application into various 
supported languages. 

static files do not require processing (e.g. images, CSS stylesheets, etc). 

ABOUT and README documents are self-explanatory. 

errors store error reports generated by the application. 

sessions store information related to each particular user. 

databases store SQLite databases and additional table information. 

cache store cached application items. 

modules are other optional Python modules. 

private files are accessed by the controllers but not directly by the 
developer. 

uploads files are accessed by the models but not directly by the developer 
(e.g., files uploaded by users of the application). 

tests is a directory for storing test scripts, fixtures and mocks. 

Models, views, controllers, languages, and static files are accessible via the 
web administration [design] interface. ABOUT, README, and errors are 
also accessible via the administration interface through the corresponding 
menu items. Sessions, cache, modules and private files are accessible to the 
applications but not via the administration interface. 

Everything is neatly organized in a clear directory structure that is replicated 
for every installed web2py application, although the user never needs to 
access the filesystem directly: 

1 init__.py ABOUT LICENSE models views 

2 controllers modules private tests cron 
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, cache errors upload sessions static 

" init .py" is an empty file which is required in order to allow Python (and 

web2py) to import the modules in the modules directory. 

Notice that the admin application simply provides a web interface to web2py 
applications on the server file system. web2py applications can also be 
created and developed from the command-line; you don't have to use the 
browser admin interface. A new application can be created manually by 
replicating the above directory structure under,e.g., "applications/newapp/" 
(or simply untar the welcome. w2p file into your new application directory). 
Application files can also be created and edited from the command-line 
without having to use the web admin interface. 

4.6 API 

Models, controllers, and views are executed in an environment where the 
following objects are already imported for us: 

Global Objects: 

j request, response, session, cache 

Internationalization: 

■ T 

Navigation: 

, redirect, HTTP 

Helpers: 

, XML, URL, BEAUTIFY 

2 

3 A, B, BODY, BR, CENTER, CODE, COL, COLGROUP, 

4 DIV, EH, EMBED, FIELDSET, FORM, HI, H2, H3, H4, H5, H6, 

5 HEAD, HR, HTML, I, IFRAHE, IMG, INPUT, LABEL, LEGEND, 

6 LI, LINK, OL, UL, META, OBJECT, OPTION, P, PRE, 
- SCRIPT, OPTGROUP, SELECT, SPAN, STYLE, 

8 TABLE, TAG, TD, TEXTAREA, TH, THEAD, TBODY, TFOOT, 

9 TITLE, TR, TT, URL, XHTML, xmlescape, embed64 
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„ CAT, MARKHIN, MENU, ON 

Forms and tables 

, SQLFORM (SQLFORM. factory, SQLFORM . g rid , SQLFORM. smartgrid) 

Validators: 

■ CLEANUP, CRYPT, ISJLPHANUHERIC, IS_DATE_IN_RANGE, IS_DATE, 

z IS_DATETIHE_IN_RANGE, IS_DATETIME, IS_DECIHAL_IN_RANGE, 

3 IS_EMAIL, IS_EHPTY_OR, IS_EXPR, IS_FLOAT_IN_RANGE, IS_IMAGE, 

4 IS_IN_DB, IS_IN_SET, IS_INT_IN_RANGE, IS_IPV4, IS_LENGTH, 

5 IS_LIST_0F, IS_L0WER, IS_HATCH, IS_EQUAL_TO, IS_NOT_EHPTY, 

6 IS_NOT_IN_DB, IS_NULL_0R, IS_SLUG, IS_STR0NG, IS_TIHE, 
- IS_UPLOAD_FILENAME, IS_UPPER, IS_URL 

Database: 

, DAL, Field 


For backward compatibility SQLDB=DAL and SQLField=Field. We encourage you 
to use the new syntax DAL and Field, instead of the old syntax. 

Other objects and modules are defined in the libraries, but they are not 
automatically imported since they are not used as often. 

The core API entities in the web2py execution environment are request, 
response, session, cache, URL, HTTP, redirect and T and are discussed below. 

A few objects and functions, including Auth, Crud and Service, are defined 
in "gluon/tools.py" and they need to be imported as necessary: 
from gluon. tools import Auth, Crud, Service 


4.6.1 Accessing the API from Python modules 

Your models or controller may import python modules, and these may need 
to use some of the web2py API. The way for them to do it is importing them: 

from gluon import * 

In fact, any Python module, even if not imported by a web2py application, 
can import the web2py API as long as web2py is in the sys . path. 
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There is one caveat, though. Web2py defines some global objects (request, 
response, session, cache, T) that can only exist when an HTTP request is 
present (or is faked). Therefore, modules can access them only if they are 
called from an application. For this reasons they are placed into a container 
caller current, which is a thread local object. Here is an example. 

Create a module "/myapp/modules/testpy" that contains: 

from gluon import * 

def ip(): return current . request. client 

Now from a controller in "myapp" you can do 


] import test 


2 def index( ) : 


3 return "Your ip is 

' + test .ip( ) 


Notice a few things: 

• import test looks for the module first in the current app's modules folder, 
then in the folders listed in sys.path. Therefore, app-level modules always 
take precendence over Python modules. This allows different apps to ship 
with different versions of their modules, without conflicts. 

• Different users can call the same action index concurrently, which calls the 
function in the module, and yet there is no conflict because cu rrent . request 
is a different object in different threads. Just be careful not to access 
current . request outside of functions or classes (i.e., at the top level) in 
the module. 

• import test is a shortcut for from applications .appname. modules import 
test. Using the longer syntax, it is possible to import modules from other 
applications. 

For uniformity with normal Python behavior, by default web2py does not 
reload modules when changes are made. Yet this can be changed. To turn on 
the auto-reload feature for modules, use the track_changes function as follows 
(typically in a model file, before any imports): 

from gluon. custom_import import track„changes; track_changes(True) 

From now on, every time a module is imported, the importer will check 
if the Python source file (py) has changed. If it has changed, the module 
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will be reloaded. This applies to all Python modules, even Python modules 
outside web2py. The mode is global and applies to all applications. Changes 
made to models, controllers, and views are always reloaded regardless of the 
mode used. To turn the mode off, use the same function with False as the 
argument. To know the actual tracking state, use the is_tracking_changes() 
function, also from gluon. custom_import. 

Modules that import current can access: 

• current. request 

• current . response 

• current . session 

• current . cache 

• current. T 

and any other variable your application chooses to store in current. For 
example a model could do 

, auth = Auth(db) 

2 from gluon import current 

3 current. auth = auth 

and now all modules imported can access 

• current. auth 

current and import create a powerful mechanism to build extensible and 
reusable modules for your applications. 

There is one major caveat. Given from gluon import current, it is correct to 
use current, request and any of the other thread local objects but one should 
never assign them to global variables in the module, such as in 

, request = current, request # WRONG! DANGER! 

nor one should use it assign class attributes 

1 class MyClass: 

request = current, request # WRONG! DANGER! 

This is because the thread local object must be extracted at runtime. Global 
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variables instead are defined only once when the model is imported for the first 
time. 


4. J request 

The request object is an instance of the ubiquitous web2py class that is called 
gluon. storage. Storage, which extends the Python diet class. It is basically a 
dictionary, but the item values can also be accessed as attributes: 

request .vars 

is the same as: 
request! ' vars' ] 

Unlike a dictionary, if an attribute (or key) does not exist, it does not raise an 
exception. Instead, it returns None. 

It is sometimes useful to create your own Storage objects. You can do so as 
follows: 

1 from gluon. storage import Storage 

2 mystorage = Storagef) # empty storage object 

3 my_other_storage = Storage(dict(a=l, b=2)) # convert dictionary to Storage 

request has the following items /attributes, some of which are also an instance 
of the Storage class: 

• request .cookies: a Cookie. SimpleCookie( ) object containing the cookies 
passed with the HTTP request. It acts like a dictionary of cookies. Each 
cookie is a Morsel object. 

• request. env: a Storage object containing the environment variables passed 
to the controller, including HTTP header variables from the HTTP request 
and standard WSGI parameters. The environment variables are all 
converted to lower case, and dots are converted to underscores for easier 
memorization. 

• request .application: the name of the requested application (parsed from 
request .env. path_info). 
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request . controller: the name of the requested controller (parsed from the 
request .en v. path_info). 

request .function: the name of the requested function (parsed from the 
request .en v. path_info). 

request .extension: the extension of the requested action. It defaults to 
"html". If the controller function returns a dictionary and does not specify 
a view, this is used to determine the extension of the view file that will 
render the dictionary (parsed from the request. env.path_info). 

request . folder: the application directory. For example if the application is 
"welcome", request . folder is set to the absolute path "/path/to/welcome". 
In your programs, you should always use this variable and the 
os. path. join function to build paths to the files you need to access. 
Although web2py always uses absolute paths, it is a good rule never to 
explicitly change the current working folder (whatever that is) since this is 
not a thread-safe practice. 

request, now: a datetime.datetime object storing the datetime of the current 
request. 

request . utcnow: a datetime . datetime object storing the UTC datetime of the 
current request. 

request .a rgs: A list of the URL path components following the controller 
function name; equivalent to request . env.path_info. split ( '/' ) [3: ] 

request . vars: a gluon. storage. Storage object containing the HTTP GET and 
HTTP POST query variables. 

request . get_vars: a gluon . storage. Storage object containing only the HTTP 
GET query variables. 

request . post- vars: a gluon . storage. Storage object containing only the 
HTTP POST query variables. 

request .client: The ip address of the client as determined by, if 
present, request, env. http_x_forwarded_for or by request .env. remote_addr 
otherwise. While this is useful it should not be trusted because the 
http_x_forwarded_for can be spoofed. 
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• request .is_local: True if the client is localhost, False otherwise. Should 
work behind a proxy if the proxy supports http_x_forwardecLfor. 

• request .is_https: True if the request is using the HTTPS protocol, False 
otherwise. 

• request .body: a read-only file stream that contains the body of the HTTP 
request. This is automatically parsed to get the request . post_vars and then 
rewinded. It can be read with request . body . read ( ) . 

• request . a j ax is True if the function is being called via an Ajax request. 

• request . cid is the id of the component that generated the Ajax request (if 
any). You can read more about components in Chapter 12. 

• request . restful this is a new and very useful decorator that can be 
used to change the default behavior of web2py actions by separating 
GET /POST /PUSH /DELETE requests. It will be discussed in some detail 
in Chapter 10. 

• request . user_agent( ) parses the user_agent field from the client and 
returns the information in the form of a dictionary. It is useful to detect 
mobile devices. It uses "gluon/contrib/user_agent_parser.py" created by 
Ross Peoples. To see what it does, try to embed the following code in a 
view: 

{{=BEAUTIFY( request . user_agent ( ) ) }} 

• request . wsgi is a hook that allows you to call third party WSGI applications 
from inside actions 

The latter includes: 

• request .wsgi. environ 

• request .wsgi. start_response 

• request .wsgi. middleware 

their usage is discussed at the end of this Chapter. 
As an example, the following call on a typical system: 

http : //127 . 0.0 . 1 : 8000/examples/def ault/status/x/y/z?p=l&q=2 
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results in the following request object: 
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variable 

value 

request.application 

examples 

request. controller 

default 

request. function 

index 

request. extension 

html 

request, view 

status 

request. folder 

applications/ examples / 

request. args 

[v, y, 'z'] 

request. vars 

<Storage {'p': l, 'q': 2}> 

request. get_vars 

<Storage {'p': l, 'q': 2}> 

request.post_vars 

<Storage ||> 

request. isjocal 

False 

request.is_https 

False 

request. ajax 

False 

request. cid 

None 

request. wsgi 

hook 

request. env.contentjength 



request. env.content_type 


request. env.http_accept 

text / xmLtext /html; 

request. env.http_accept_encoding 

gzip, deflate 

request. env.http_accept_language 

en 

request. env.http_cookie 

session_id_examples= 127.0.0. 1 .1 19725 

request. env.http_host 

127.0.0.1:8000 

request. env.http_max_forwards 

10 

request. env.http_referer 

http://web2py.c0m/ 

request. env.http_user_agent 

Mozilla/5.0 

request. env.http_via 

1.1 web2py.c0m 

request. env.http_x_forwarded_for 

76.224.34.5 

request.env.http_x_forwarded_host 

web2py.c0m 

request. env.http_x_forwarded_server 

127.0.0.1 

request. env.path_info 

/examples / simple_examples / status 

request. env.query_string 

remote_addr:i27.o.o.i 

request. env.request_method 

GET 

request. env.script_name 


request. env.server_name 

127.0.0.1 

request.env.server_port 

8000 

request.env.server_protocol 

HTTP/1.1 

request.env.web2py_path 

/Users/mdipierro/web2py 

request. env.web2py_version 

Version 1.99.1 

request. env.web2py_runtime_gae 

(optional, defined only if GAE detected) 

request. env.wsgi_errors 

<open file, mode 'w' at > 

request. env.wsgi_input 


request. env.wsgi_multiprocess 

False 

request. env.wsgi_multi thread 

True 

request.env.wsgi_run_once 

False 

request.env.wsgi_url_scheme 

http 

request. env.wsgi_version 

10 
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Which environment variables are actually defined depends on the web server. 
Here we are assuming the built-in Rocket wsgi server. The set of variables is 
not much different when using the Apache web server. 

The re quest. env.http_* variables are parsed from the request HTTP header. 

The request .env.web2py_* variables are not parsed from the web server 
environment, but are created by web2py in case your applications need to 
know about the web2py location and version, and whether it is running on 
the Google App Engine (because specific optimizations may be necessary). 

Also notice the request. env.wsgi_* variables. They are specific to the wsgi 
adaptor. 

4.8 response 

response is another instance of the Storage class. It contains the following: 

response, body: a StringlO object into which web2py writes the output page 
body. NEVER CHANGE THIS VARIABLE. 

response, cookies: similar to request . cookies, but while the latter contains the 
cookies sent from the client to the server, the former contains cookies sent by 
the server to the client. The session cookie is handled automatically. 

response. download ( request , db): a method used to implement the controller 
function that allows downloading of uploaded files, request . download expects 
the last arg in request. a rgs to be the encoded filename (i.e., the filename 
generated at upload time and stored in the upload field). It extracts the 
upload field name and table name as well as the original filename from 
the encoded filename, response. download takes two optional arguments: 
chunk_size sets the size in bytes for chunked streaming (defaults to 64K), and 
attachments determines whether the downloaded file should be treated as an 
attachment or not (default to True). Note, response. download is specifically for 
downloading files associated with db upload fields. Use response, stream (see 
below) for other types of file downloads and streaming. Also, note that it is 
not necessary to use response. download to access files uploaded to the /static 
folder - static files can (and generally should) be accessed directly via URL 
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(e.g., /app/ static /files/ myfile.pdf). 

response. files: a list of CSS and JS files required by the page. They will 
automatically be linked in the head of the standard "layout.html" via the 
included "web2py_ajax.html". To include a new CSS or JS file, just append it 
to this list. It will handle duplicates. The order is significant. 

response. include_files() generates html head tags to includes all 
response, files (used in "views/ web2py_ajax.html"). 

response. flash: optional parameter that may be included in the views. 
Normally used to notify the user about something that happened. 

response, headers: a diet for HTTP response headers. Web2py sets some 
headers by default, including "Content-Length", "Content-Type", and "X- 
Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", 
"Expires", and "Pragma" headers to prevent client-side caching, except for 
static file requests, for which client-side caching is enabled. The headers that 
web2py sets can be overwritten or removed, and new headers can be added 
(e.g., response, headers [ 'Cache-Control ' ] = 'private'). 

response. menu: optional parameter that may be included in the views, 
normally used to pass a navigation menu tree to the view. It can be rendered 
by the MENU helper. 

response. meta: a storage object (like a diet) that contains optional meta 
information like response. meta. author, .description, and/or .keywords. The 
content of each meta variable is automatically placed in the proper META tag 
by the code in "views/web2py _ajax.html", which is included by default in 
"views / layout.html" . 

response. include_meta ( ) generates a string that includes all response. meta 
headers serialized (used in "views/web2py _ajax.html"). 

response, postprocessing: this is a list of functions, empty by default. These 
functions are used to filter the response object at the output of an action, 
before the output is rendered by the view. It can be used to implement 
support for other template languages. 
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response. render(view, vars): a method used to call the view explicitly inside 
the controller, view is an optional parameter which is the name of the view 
file, vars is a dictionary of named values passed to the view. 

response. session_f ile: file stream containing the session. 

response. session_file_name: name of the file where the session will be saved. 

response. session_id: the id of the current session. It is determined 
automatically. NEVER CHANGE THIS VARIABLE. 

response. session_icLname: the name of the session cookie for this application. 
NEVER CHANGE THIS VARIABLE. 

response. status: the HTTP status code integer to be passed to the response. 
Default is 200 (OK). 

response. stream(file, chunk_size, request=request): when a controller 
returns it, web2py streams the file content back to the client in blocks of 
size chunk_size. The request parameter is required to use the chunk start 
in the HTTP header. As noted above, response. download should be used to 
retrieve files stored via an upload field, response, stream can be used in other 
cases, such as returning a temporary file or StringlO object created by the 
controller. Unlike response. download, response, stream does not automatically 
set the Content-Disposition header, so you may need to do that manually 
(e.g., to specify download as an attachment, and to provide a filename). 
However, it does automatically set the Content -Type header (based on the 
filename extension). 

response, subtitle: optional parameter that may be included in the views. It 
should contain the subtitle of the page. 

response. title: optional parameter that may be included in the views. It 
should contain the title of the page and should be rendered by the HTML 
title TAG in the header. 

response. toolbar: a function that allows you embed a toolbar into page form 
debugging purposes {{=response.toolbar()}}. The toolbar displays request, 
response, session variables and database access time for each query. 
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response. _vars: this variable is accessible only in a view, not in the action. It 
contains the value returned by the action to the view. 

response. optimize_css: if can be set to "concat,minifyinline" to concatenate, 
minity and inline the CSS files included by web2py 

response, optimizers: if can be set to "concat,minify,inline" to concatenate, 
minity and inline the CSS files included by web2py 

response . view: the name of the view template that must render the page. This 

is set by default to: 

"%s/%s.%s" % (request .controller, request . function, request .extension) 

or, if the above file cannot be located, to 

"generic. %s" % (request .extension) 

Change the value of this variable to modify the view file associated with a 
particular action. 

response. xmlrpc( request, methods): when a controller returns it, this function 
exposes the methods via XML-RPC [46]. This function is deprecated since a 
better mechanism is available and described in Chapter 10. 

response. write(text): a method to write text into the output page body. 

response, js can contain Javascript Code. This code will be executed if and 
only if the response is received by a web2py component as discussed in 
Chapter 12. 

Since response is a gluon . storage. Storage object, it can be used to store other 
attributes that you may want to pass to the view. While there is no technical 
restriction, our recommendation is to store only variables that are to be 
rendered by all pages in the overall layout ("layout.html"). 



Anyway, we strongly suggest to stick to the variables listed here: 

1 

response. title 

2 

response. subtitle 

3 

response. flash 

4 

response. menu 

5 

response . met a .author 

6 

response . meta . desc ription 

7 

response. meta. keywords 
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8 response. meta.* 


because this will make it easier for you to replace the standard "layout.html" 
file that comes with web2py with another layout file, one that uses the same 
set of variables. 

Old versions of web2py user response. author instead of response. meta. author 
and similar for the other meta attributes. 


4-9 


session is another instance of the Storage class. Whatever is stored into 
session for example: 
session. myvariable = "hello" 

can be retrieved at a later time: 

a = session. myvariable 

as long as the code is executed within the same session by the same user 
(provided the user has not deleted session cookies and the session did not 
expire). Because session is a Storage object, trying to access an attribute/key 
that has not been set does not raise an exception; it returns None instead. 

The session object has three important methods. One is forget: 
session . forget ( response) 

It tells web2py not to save the session. This should be used in those 
controllers whose actions are called often and do not need to track user 
activity. session. forget () prevents the session file from being written, 
regardless of whether it has been modified. session, forget (response) 
additionally unlocks and closes the session file. You rarely need to call this 
method since sessions are not saved when they are not changed. However, if 
the page makes multiple simultaneous Ajax requests, it is a good idea for the 
actions called via Ajax to call session, forget (response) (assuming the session 
is not needed by the action). Otherwise, each Ajax action will have to wait for 
the previous one to complete (and unlock the session file) before proceeding, 
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which will slow down the page loading. Notice that sessions are not locked 
when stored in database. 

Another method is: 
session. secure( ) 

which tells web2py to set the session cookie to be a secure cookie. This 
should be set if the app is going over https. By setting the session cookie 
to be secure, the server is asking the browser not to send the session cookie 
back to the server unless over an https connection. 

The other method is connect: 

session. connect (request, response, db, masterapp=None) 

where db is the name of an open database connection (as returned by the 
DAL). It tells web2py that you want to store the sessions in the database 
and not on the filesystem. session . connect must come after db=DAL( . . . ), but 
before any other logic that requires session, for example, setting up Auth. 
web2py creates a table: 


1 

db.define_table( 

l vieb2pysession' , 

2 


Field (' locked' , 'boolean', default=False) , 

3 


Field ( ' client-ip' ) , 

4 


Field ( ' created_datetime' , 'datetime', default=now) , 

5 


Field ( ' modified-datetiise' , 'datetime' ) , 

6 


Field ( ' unique_key ' ) , 

7 


Field (' session-data' , 'text')) 


and stores cPickled sessions in the session_data field. 

The option masterapp=None, by default, tells web2py to try to retrieve an 
existing session for the application with name in request. application, in the 
running application. 

If you want two or more applications to share sessions, set masterapp to the 
name of the master application. 

You can check the state of your application at any time by printing the 
request, session and response system variables. One way to do it is to create 
a dedicated action: 
def status () : 
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return diet (request=request : session-session, response=response) 


4..9.1 Separate sessions 

If you are storing sessions on filesystems and you have lots of them, the file 
system access may become a bottle-neck. One solution is the following: 

session. connect (request, response, separate=True) 

By setting separate=True web2py will store sessions not in the "sessions/" 
folder but in subfolders of the "sessions/" folder. The subfolder will be 
created automatically. Sessions with the same prefix will be in the same 
subfolder. Again, note that the above must be called before any logic that 
might require the session. 


4.20 cache 

cache a global object also available in the web2py execution environment. It 
has two attributes: 

• cache, ram: the application cache in main memory. 

• cache. disk: the application cache on disk. 

cache is callable, this allows it to be used as a decorator for caching actions 
and views. 

The following example caches the time. ctime( ) function in RAM: 


def 

cache_in_ram( ) : 


import time 


t = cache. ram (' time' , lambda: time.ctime( ) , time_expire=5) 


return dict(time=t, link=A( ' click me' , _href=request .url) ) 


The output of lambda: time.ctimeO is cached in RAM for 5 seconds. The 
string 'time' is used as cache key. 

The following example caches the time. ctime( ) function on disk: 
1 def cache_on_disk( ) : 
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2 import time 

3 t = cache. disk( ' time' , lambda: time.ctime( ) , time_expire=5) 

4 return dict(time=t, link=A( 'click me' , _href=request . url) ) 

The output of lambda: time. ctime( ) is cached on disk (using the shelve 
module) for 5 seconds. 

Note, the second argument to cache . ram and cache . disk must be a function or 
callable object. If you want to cache an existing object rather than the output 
of a function, you can simply return it via a lambda function: 

1 cache. ram ( 'myobject' , lambda: myobject, time_expire=60*6O*24) 

The next example caches the time . ctime ( ) function to both RAM and disk: 

1 def cache_in_ram_and_disk( ) : 

2 import time 

3 t = cache. ram (' time' , lambda: cache. disk( ' time' , 

4 lambda: time.ctime( ) , time_expire=5) , 

5 time_expire=5) 

6 return dict(time=t, link=A( 'click me' , _href=request . url) ) 

The output of lambda: time, ctime () is cached on disk (using the shelve 
module) and then in RAM for 5 seconds. web2py looks in RAM first 
and if not there it looks on disk. If it is not in RAM or on disk, lambda: 
time.ctimeO is executed and the cache is updated. This technique is useful 
in a multiprocess environment. The two times do not have to be the same. 

The following example is caching in RAM the output of the controller 
function (but not the view): 

1 @cache( request .env.path_info : time_expire=5 : cache_model=cache. ram) 

2 def cache_controller_in_ram( ) : 

3 import time 

4 t = time.ctimeO 

5 return dict(time=t, link=A( 'click me' , _href=request . url) ) 

The dictionary returned by cache_controller_in_ram is cached in RAM for 5 
seconds. Note that the result of a database select cannot be cached without 
first being serialized. A better way is to cache the database select directly 
using the select method cache argument. 

The following example is caching the output of the controller function on 
disk (but not the view): 
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The dictionary returned by cache_controller_on_disk is cached on disk for 
5 seconds. Remember that web2py cannot cache a dictionary that contains 
un-pickleable objects. 

It is also possible to cache the view. The trick is to render the view in the 
controller function, so that the controller returns a string. This is done by 
returning response, render(d) where d is the dictionary we intended to pass to 
the view. The following example caches the output of the controller function 
in RAM (including the rendered view): 



response, render(d) returns the rendered view as a string, which is now 
cached for 5 seconds. This is the best and fastest way of caching. 

Note, time_expire is used to compare the current time with the time the 
requested object was last saved in the cache. It does not affect future requests. 
This enables time_expire to be set dynamically when an object is requested 
rather than being fixed when the object is saved. For example: 

message = cache. ram ( 'message' , lambda: 'Hello', time_expire=5) 

Now, suppose the following call is made 10 seconds after the above call: 

message = cache. ram ( 'message' , lambda: 'Goodbye' , time_expire=20) 

Because time_expire is set to 20 seconds in the second call and only 10 seconds 
has elapsed since the message was first saved, the value "Hello" will be 
retrieved from the cache, and it will not be updated with "Goodbye". The 
time_expire value of 5 seconds in the first call has no impact on the second 
call. 
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Setting time_expire=0 (or a negative value) forces the cached item to be 
refreshed (because the elapsed time since the last save will always be > o), 
and setting time_expire=None forces retrieval of the cached value, regardless of 
the time elapsed since it was saved (if time_expire is always None, the cached 
item will effectively never expire). 

You can clear one or more cache variables with 
cache. ram. clear ( regex=' . . . ' ) 

where regex is a regular expression matching all the keys you want removed 
from the cache. You can also clear an single item with: 
cache. ram (key, None) 

where key is the key of the cached item. 

It is also possible to define other caching mechanisms such as memcache. 
Memcache is available via gluon.cont rib. memcache and is discussed in more 
details in Chapter 14. 

Be careful when caching that caching is usually at the app-level not at the user 
level. If you need, for eaxmple, to cache user specific content, choose a key that 
includes the user id. 


4..II URL 

The URL function is one of the most important functions in web2py It 
generates internal URL paths for the actions and the static files. 

Here is an example: 
URL('f') 

is mapped into 

/ [application ]/[cont roller] /f 

Notice that the output of the URL function depends on the name of the 
current application, the calling controller, and other parameters. web2py 
supports URL mapping and reverse URL mapping. URL mapping allows 
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you to redefine the format of external URLs. If you use the URL function to 
generate all the internal URLs, then additions or changes to URL mappings 
will prevent broken links within the web2py application. 

You can pass additional parameters to the URL function, i.e., extra terms in 
the URL path (args) and URL query variables (vars): 
URL('f' , args=['x', 'y'], vars=dict(z=' t ' ) ) 

is mapped into 

/ [application ]/[controller]/f/x/y?z=t 

The args attributes are automatically parsed, decoded, and finally stored in 
request. args by web2py. Similarly, the vars are parsed, decoded, and then 
stored in request .vars. args and vars provide the basic mechanism by which 
web2py exchanges information with the client's browser. 

If args contains only one element, there is no need to pass it in a list. 

You can also use the URL function to generate URLs to actions in other 

controllers and other applications: 

URL('a', ' c ' , 'f, args=['x', 'y'], vars=dict (z=' t ' ) ) 

is mapped into 

/a/c/f/x/y?z=t 

It is also possible to specify application, controller and function using named 
arguments: 


URL(a='a', c='C, f='f) 

If the application name a is 

missing the current app 

is assumed. 

URLCC, •f) 

If the controller name is missing, the current one is 

assumed. 

URL('f' ) 


Instead of passing the name of a controller function it is also possible to pass 
the function itself 
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URL(f) 


For the reasons mentioned above, you should always use the URL function to 
generate URLs of static files for your applications. Static files are stored in the 
application's static sub folder (that's where they go when uploaded using the 
administrative interface). web2py provides a virtual 'static' controller whose 
job is to retrieve files from the static sub folder, determine their content-type, 
and stream the file to the client. The following example generates the URL 
for the static file "image. png": 

URL( 'static' , ' image. png' ) 

is mapped into 

/[application] /st a tic/image. png 

If the static file is in a sub folder within the static folder, you can include the 
subfolder(s) as part of the filename. For example, to generate: 

/[application] /static/images/icons/arrow. png 

one should use: 

URL( 'static' , ' images/ icons/ arrow. png' ) 

You do not need to encode/escape the args and vars arguments; this is done 
automatically for you. 

By default, the extension corresponding to the current request (which 
can be found in request .extension) is appended to the function, unless 
request. extension is html, the default. This can be overridden by explicitly 
including an extension as part of the function name URL ( f = ' name . ext ' ) or with 
the extension argument: 

URL(..., extension^ ess ' ) 

The current extension can be explicitly suppressed: 
URL(..., extension=False) 
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4.11.1 Absolute urls 

By default, URL generates relative URLs. However, you can also generate 
absolute URLs by specifying the scheme and host arguments (this is useful, 
for example, when inserting URLs in email messages): 
1 URL(..., scheme='/?rrp' , host= ' www. mysite. com' ) 

You can automatically include the scheme and host of the current request by 
simply setting the arguments to True. 
] URL(..., scheme=True, host=True) 

The URL function also accepts a port argument to specify the server port if 
necessary. 

4..11.2 Digitally signed urls 

When generating a URL, you have the option to digitally sign it. This will 
append a -signature GET variable that can be verified by the server. This can 
be done in two ways. 

You can pass to the URL function the following arguments: 

• hmac_key: the key for signing the URL (a string) 

• salt: an optional string to salt the data before signing 

• hash_vars: an optional list of names of variables from the URL query string 
(i.e., GET variables) to be included in the signature. It can also be set to 
True (the default) to include all variables, or False to include none of the 
variables. 

Here is an example of usage: 

, KEY = 'mykey' 
2 

3 def one( ) : 

4 return diet (link=URL( ' two' , vars=dict (a=123) , hmac_key=KEY) 
5 

6 def two( ) : 

- if not URL.verify(hmac_key=KEY): raise HTTP(403) 

8 # do something 
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return locals( ) 


This makes the action two accessible only via a digitally signed URL. A 
digitally signed URL looks like this: 

, ' /welcome/ default/ two?a=123&_signature=4981bc70el3866bb60e52a0907356eae822224e9' 

Note, the digital signature is verified via the URL. verify function. URL. verify 
also takes the hmac_key, salt, and hash_vars arguments described above, and 
their values must match the values that were passed to the URL function when 
the digital signature was created in order to verify the URL. 

A second and more sophisticated but more common use of digitally signed 
URLs is in conjunction with Auth. This is best explained with an example: 

] @auth. requires_login( ) 

z def one( ) : 

3 return diet (link=URL( ' two' , vars=dict (a=123) , user_signature=True) 

4 

5 @auth. requires_signature( ) 

6 def two() : 

7 # do something 

s return locals( ) 

In this case the hmac_key is automatically generated and shared within the 
session. This allows action two to delegate any access control to action one. If 
the link is generated and signed, it is valid; else it is not. If the link is stolen 
by another user, the link will be invalid. 

It is good practice to always digitally sign Ajax callbacks. If you use the 
web2py LOAD function, it has a user_signature argument too that can be used 
for this purpose: 
] {{=L0AD( 'default ' , 'two', vars=dict(a=123) , ajax=True, user_signature=True)}} 


4.22 HTTP flnd redirect 

web2py defines only one new exception called HTTP. This exception can be 
raised anywhere in a model, a controller, or a view with the command: 
raise HTTP(400 : "my message") 
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It causes the control flow to jump away from the user's code, back to web2py, 
and return an HTTP response like: 

, HTTP/1.1 400 BAD REQUEST 

2 Date: Sat, 05 Jul 2008 19:36:22 GMT 

3 Server: Rocket WSGI Server 

4 Content-Type: text/html 

5 Via: 1.1 127.0.0.1:8000 

6 Connection: close 

7 Transfer-Encoding: chunked 

8 

i) my message 

The first argument of HTTP is the HTTP status code. The second argument 
is the string that will be returned as the body of the response. Additional 
optional named arguments are used to build the response HTTP header. For 
example: 

1 raise HTTP(400, 'my message', test='heIIo') 

generates: 

, HTTP/1.1 400 BAD REQUEST 

2 Date: Sat, 05 Jul 2008 19:36:22 GMT 

3 Server: Rocket WSGI Server 

4 Content-Type: text/html 

5 Via: 1.1 127.0.0.1:8000 

6 Connection: close 

7 Transfer-Encoding: chunked 

8 test: hello 

10 my message 


If you do not want to commit the open database transaction, rollback before 
raising the exception. 

Any exception other than HTTP causes web2py to roll back any open database 
transaction, log the error traceback, issue a ticket to the visitor, and return a 
standard error page. 

This means that only HTTP can be used for cross-page control flow. Other 
exceptions must be caught by the application, otherwise they are ticketed by 
web2py. 

The command: 
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] redirect ( 'http://www.web2py.com' ) 
is simply a shortcut for: 

■ raise HTTP(303, 

z 'You are being redirected <a href="%s">here</a>' % location, 

3 Location^' http://www.web2py.com' ) 

The named arguments of the HTTP initializer method are translated into 
HTTP header directives, in this case, the redirection target location, redirect 
takes an optional second argument, which is the HTTP status code for the 
redirection (303 by default). Change this number to 307 for a temporary 
redirect or to 301 for a permanent redirect. 

The most common way to use redirect is to redirect to other pages in the 
same app and (optionally) pass parameters: 

1 redirect ( URL ( 'index' , args=(l,2,3) , vars=dict (a='o' ) ) ) 


4.1 3 j and Internationalization 

The object T is the language translator. It constitutes a single global instance 
of the web2py class gluon. language, translator. All string constants (and only 
string constants) should be marked by T, for example: 

1 a = T( "hello world" ) 

Strings that are marked with T are identified by web2py as needing language 
translation and they will be translated when the code (in the model, 
controller, or view) is executed. If the string to be translated is not a constant 
but a variable, it will be added to the translation file at runtime (except on 
GAE) to be translated later. 

The T object can also contain interpolated variables and support multiple 
equivalent syntax: 

, a = T( "hello %s", ('Tim' ,)) 

2 a = T("hello %(name)s" , dict(name=' Tim' ) ) 

3 a = J ("hello %s") % ('Tim',) 

4 a = T("hello %(name)s") % diet (name=' Tim' ) 
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The latter syntax is recommended because it makes translation easier. The 
first string is translated according to the requested language file and the name 
variable is replaced independently of the language. 

Concatenating translation translated strings and normal strings is possible: 
7{"blah ") + name + T(" blah") # invalid! 

but the opposite no: 

name + T(" blah") # invalid! 

The following code is also allowed and often preferable: 

TC'blah %(name)s blah", diet (name=' Tim' ) ) 

or the alternative syntax 

TC'blah %(name)s blah") % diet (name=' Tim' ) 

In both cases the translation occurs before the variable name is substituted in 
the "%(name)s" slot. The following alternative should NOT BE USED: 

TC'blah %(name)s blah" % diet (name=' Tim' ) ) 

because translation would occur after substitution. 

The requested language is determined by the "Accept-Language" field in 
the HTTP header, but this selection can be overwritten programmatically 
by requesting a specific file, for example: 
T.force('it-it') 

which reads the "languages/it-it.py" language file. Language files can be 
created and edited via the administrative interface. 

You can also force a per-string language: 

TC'Hello World", language="it-it" ) 

You can turn off translations completely via 

T.force(None) 

Normally, string translation is evaluated lazily when the view is rendered; 
hence, the translator force method should not be called inside a view. 
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It is possible to disable lazy evaluation via 

] T.lazy = False 

In this way, strings are translated immediately by the T operator based on the 
currently accepted or forced language. 

It is also possible to disable lazy evaluation for individual strings: 
1 T("HeIIo World", lazy=False) 

A common issue is the following. The original application is in English. 
Suppose that there is a translation file (for example Italian, "it-it. py") and the 
HTTP client declares that it accepts both English (en) and Italian (it-it) in that 
order. The following unwanted situation occurs: web2py does not know the 
default is written in English (en). Therefore, it prefers translating everything 
into Italian (it-it) because it only found the Italian translation file. If it had 
not found the "it-it.py" file, it would have used the default language strings 
(English). 

There are two solutions for this problem: create a translation language for 
English, which would be redundant and unnecessary, or better, tell web2py 
which languages should use the default language strings (the strings coded 
into the application). This can be done with: 
] T. set_current_languages( 'en' , 'en-en') 

It stores in T.current_languages a list of languages that do not require 
translation and forces a reload of the language files. 

Notice that "it" and "it-it" are different languages from the point of view of 
web2py. To support both of them, one would need two translation files, 
always lower case. The same is true for all other languages. 

The currently accepted language is stored in 

1 T.accepted_language 

Mind that T(...) does not just translate strings but can also translated 
variables: 

, >» a="test" 

2 >» print T(a) 
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In this case the word "test" in translated but, if not found and if the filesystem 
is writable, it will add it to the list of words to be translated in the language 
file. 


4.24 Cookies 

web2py uses the Python cookies modules for handling cookies. 

Cookies from the browser are in re quest. cookies and cookies sent by the 
server are in response, cookies. 

You can set a cookie as follows: 

1 response. cookies! 'mycookie' ] = ' somevalue' 

2 response. cookies! ' mycookie' ][ 'expires ' ] = 24 * 3600 

3 response. cookies! 'mycookie' ][ 'path' ] = '/' 

The second line tells the browser to keep the cookie for 24 hours. The third 
line tells the browser to send the cookie back to any application (URL path) 
at the current domain. 

The cookie can be made secure with: 
] response. cookies! 'mycookie' ][ 'secure' ] = True 

This tells the browser only to send the cookie back over HTTPS and not over 
HTTP. 

The cookie can be retrieved with: 

1 if request . cookies. has_ key ( 'mycookie' ) : 

2 value = request .cookies! 'mycookie' ] .value 

Unless sessions are disabled, web2py, under the hood, sets the following 
cookie and uses it to handle sessions: 

1 response. cookies [response. session_id_name] = response. session_id 

2 response. cookies [response. session_id_name] [ 'path' ] = "/" 

Note, if a single application includes multiple subdomains, and you want 
to share the session across those subdomains (e.g., subi.yourdomain.com, 
sub2.yourdomain.c0m, etc.), you must explicitly set the domain of the session 
cookie as follows: 
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if not request .env. remote_addr in [' 127.0.0.1' , ' localhost' ]: 

response. cookies [response. session_id_name] [ 'domain ' ] = " .yourdomain. com" 

The above can be useful if, for example, you want to allow the user to remain 
logged in across subdomains. 


4.25 Application init 

When you deploy web2py, you will want to set a default application, i.e., the 
application that starts when there is an empty path in the URL, as in: 

http://127. 0.0. 1:8000 

By default, when confronted with an empty path, web2py looks for an 
application called init. If there is no init application it looks for an application 
called welcome. 

The name of the default application can be changed from init to another 
name by setting def ault_application in routes. py: 

default_application = "myapp" 

Note: default-application first appeared in web2py version 1.83. 
Here are four ways to set the default application: 

• Call your default application "init". 

• Set default_application to your application's name in routes. py 

• Make a symbolic link from "applications /init" to your application's folder. 

• Use URL rewrite as discussed in the next section. 

4.26 URL rewrite 

web2py has the ability to rewrite the URL path of incoming requests prior 
to calling the controller action (URL mapping), and conversely, web2py can 
rewrite the URL path generated by the URL function (reverse URL mapping). 
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One reason to do this is for handling legacy URLs, another is to simplify 
paths and make them shorter. web2py includes two distinct URL rewrite 
systems: an easy-to-use parameter-based system for most use cases, and a 
flexible pattern-based system for more complex cases. To specify the URL 
rewrite rules, create a new file in the "web2py" folder called routes. py (the 
contents of routes. py will depend on which of the two rewrite systems you 
choose, as described in the next two sections). The two systems cannot be 
mixed. 

Notice that if you edit routes. py, you must reload it. This can be done in two 
ways: by restarting the web server or by clicking on the routes reload button 
in admin. If there is a bug in routes, they will not reload. 

4..16.1 Parameter-based system 

The parameter-based (parametric) router provides easy access to several 
"canned" URL-rewrite methods. Its capabilities include: 

* Omitting default application, controller and function names from 
externally-visible URLs (those created by the URL() function) 

* Mapping domains (and/or ports) to applications or controllers 

* Embedding a language selector in the URL 

* Removing a fixed prefix from incoming URLs and adding it back to 
outgoing URLs 

* Mapping root files such as /robots. txt to an applications static directory 

The parametric router also provides somewhat more flexible validation of 
incoming URLs. 

Suppose you've written an application called myapp and wish to make it the 
default, so that the application name is no longer part of the URL as seen by 
the user. Your default controller is still default, and you want to remove its 
name from user-visible URLs as well. Here's what you put in routes . py: 

routers = dict( 
BASE = diet (default_application='myapp' ) , 
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That's it. The parametric router is smart enough to know how to do the right 
thing with URLs such as: 
] http: //domain . com/myapp/default/myapp 

or 
1 http : //domain . com/myapp/myapp/index 

where normal shortening would be ambiguous. If you have two applications, 
myapp and myapp2, you'll get the same effect, and additionally myapp2's default 
controller will be stripped from the URL whenever it's safe (which is mostly 
all the time). 

Here is another case: suppose you want to support URL-based languages, 
where your URLs look like this: 
] http://myapp/en/some/path 

or (rewritten) 
1 http://en/some/path 

Here's how: 

1 routers = dict( 

2 BASE = diet (default_application='myapp' ) , 

, myapp = diet (languages=[ 'en' , 'it', 'jp'], default. language= 'en' ) , 
4 ) 

Now an incoming URL like this: 
1 http : /domain. com/it/some/path 

will be routed to /myapp/some/path, and request. uri_language will be set to 'it', 
so you can force the translation. You can also have language-specific static 
files. 

] http://domain . com/it/static/filename 

will be mapped to: 

1 applications/myapp/static/it/ filename 

if that file exists. If it doesn't, then URLs like: 
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http: //domain . com/it/static/base. ess 


will still map to: 

applications/myapp/static/base . ess 

(because there is no static/it/base. ess). 

So you can now have language-specific static files, including images, if you 
need to. Domain mapping is supported as well: 



does what you'd expect. 


routers = dict( 


BASE = dict( 


domains = { 


'domain. com: 80' 

: ' app/ insecure' , 

'domain. com: 443' 

: ' app/ secure' , 

} 
), 
) 



maps http://domain.com accesses to the controller named insecure, while 
HTTPS accesses go to the secure controller. Alternatively, you can map different 
ports to different apps, in the obvious way. 

For further information, please consult the file route r . example . py provided in 
the base folder of the standard web2py distribution. 

Note: The parameter-based system first appeared in web2py version 1.92.1. 


4.26.2 Pattern-based system 

Although the parameter-based system just described should be sufficient for 
most use cases, the alternative pattern-based system provides some additional 
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flexibility for more complex cases. To use the pattern-based system, instead 
of defining routers as dictionaries of routing parameters, you define two lists 
(or tuples) of 2-tuples, routes_in and routes_out. Each tuple contains two 
elements: the pattern to be replaced and the string that replaces it. For 
example: 

routes_in = ( 

( ' /testme' , ' /examples/ default/ index' ) , 
) 
routes_out = ( 

( ' /examples/ default/ index' , '/testme' ) , 


6 ) 


With these routes, the URL: 

http : //127 . . . 1 : 8000/testme 

is mapped into: 

http : //127 . . . 1 : 8000/examples/def autt/index 

To the visitor, all links to the page URL looks like /testme. 

The patterns have the same syntax as Python regular expressions. For 
example: 

( ' . *l .php' , '/ init/ default/ index' ) , 

maps all URLs ending in ".php" to the index page. 

Sometimes you want to get rid of the application prefix from the URLs 
because you plan to expose only one application. This can be achieved with: 

routes_in = ( 

( ' / (?P<any>. *) ' , ' /init/\g<any>' ) , 
) 
routes_out = ( 

( ' /init/ (?P<any>. *) ' , ' /\g<any>' ) , 


There is also an alternative syntax that can be mixed with the regular 
expression notation above. It consists of using $name instead of (?P<name>\w+) 
or \g<name>. For example: 

routes_in = ( 
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3 


{'/$c/$f, •/init/$c/$f), 

routes_out = ( 

C/init/$c/$f , '/$c/$f), 
) 

would also eliminate the "/example" application prefix in all URLs. 

Using the $name notation, you can automatically map routes_in to routes_out, 
provided you don't use any regular expressions. For example: 


routes_in = 

( 



C/$c/$f , 
) 

■/init/$c/$f), 



routes.out = 

[(x, y) for (y, 

x) 

in routes_in] 


If there are multiple routes, the first to match the URL is executed. If no 
pattern matches, the path is left unchanged. 

You can use $anything to match anything (.*) until the end of the line. 

Here is a minimal "routes. py" for handling favicon and robots requests: 

1 routes_in = ( 

2 (' /favicon. ico' , ' /examples/ static/ favicon. ico' ) , 

3 (' /robots. txt' , ' /examples/ static/ robots, txt' ) , 

4 ) 

5 routes_out = () 

Here is a more complex example that exposes a single app "myapp" without 
unnecessary prefixes but also exposes admin, appadmin and static: 

1 routes_in = ( 

z (' /admin/$anything' , ' /admin/ $anything') , 

i (' /static/$anything' , ' I myapp/ static/ $anything' ) , 

4 ( ' / appadmin/ $anything ' , ' /myapp/ appadmin/ $anything ' ) , 

5 (' /favicon. ico' , ' /myapp/ static/ favicon. ico' ) , 

6 {'/ robots.txt' , ' /myapp/ static/ robots. txt' ) , 

7 ) 

8 routes_out = [(x, y) for (y, x) in routes_in[ : -2] ] 

The general syntax for routes is more complex than the simple examples we 
have seen so far. Here is a more general and representative example: 

i routes_in = ( 
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z ( '1401 .191\ .\d+\. \d+:https:/ /www. web2py.com: POST /(?P<any>.*) \ .php' 
3 ' /test/ default/ index?vars=\g<any>' ) , 


It maps https POST requests to host www.web2py . com from a remote IP matching 
the regular expression 

'2401.I9I\. \d+\. \d+' 

requesting a page matching the regular expression 

1 / (?P<any>. *) \ .php' 

into 

1 /test /default / index?va rs=\ g<any> ' 

where \g<any> is replaced by the matching regular expression. 

The general syntax is 

'[remote address] : [protocol] :// [host] : [method] [path]' 

The entire expression is matched as a regular expression, so "." should 
always be escaped and any matching subexpression can be captured using 
(?P<. ..>...) according to Python regex syntax. 

This allows to reroute requests based on the client IP address or domain, 
based on the type of the request, on the method, and the path. It also 
allows web2py to map different virtual hosts into different applications. Any 
matched subexpression can be used to build the target URL and, eventually, 
passed as a GET variable. 

All major web servers, such as Apache and lighttpd, also have the ability to 
rewrite URLs. In a production environment that may be an option instead of 
routes.py. Whatever you decide to do we strongly suggest that you do not 
hardcode internal URLs in your app and use the URL function to generate 
them. This will make your application more portable in case routes should 
change. 

Application-Specific URL rewrite 

When using the pattern-based system, an application can set its own routes 
in an application-specific routes.py file located in the applications base folder. 
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This is enabled by configuring routes_app in the base routes. py to determine 
from an incoming URL the name of the application to be selected. When 
this happens, the application-specific routes. py is used in place of the base 
routes. py. 

The format of routes_app is identical to routes_in, except that the replacement 
pattern is simply the application name. If applying routes_app to the 
incoming URL does not result in an application name, or the resulting 
application-specific routes.py is not found, the base routes. py is used as 
usual. 

Note: routes_app first appeared in web2py version 1.83. 

Default application, controller, and function 

When using the pattern-based system, the name of the default application, 
controller, and function can be changed from init, default, and index 
respectively to another name by setting the appropriate value in routes.py: 


1 

default. 

.application 

= "myapp" 

2 

default. 

.controller = 

= "admin" 

3 

default. 

.function = 

•start" 


5 

6 ] 


Note: These items first appeared in web2py version 1.83. 


4.27 Routes on error 

You can also use routes . py to re-route requests to special actions in case there 
is an error on the server. You can specify this mapping globally, for each app, 
for each error code, or for each app and error code. Here is an example: 

routes_onerror = [ 

( ' init/400' , '/init/default/login' ) , 
('init/* 1 , ' /init/ static/ fail. html' ) , 
C*/404' , '/init/ static/ cant find, html') , 
('*/*', ' /init/ error/ index' ) 


For each tuple, the first string is matched against "[app name]/[error code]". 
If a match is found, the failed request is re-routed to the URL in the second 
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string of the matching tuple. If the error handling URL is a not a static file, 
the following GET variables will be passed to the error action: 

• code: the HTTP status code (e.g., 404, 500) 

• ticket: in the form of "[app name]/[ticket number]" (or "None" if no ticket) 

• requestecLuri: equivalent to request .env. request_uri 

• request_url: equivalent to request, url 

These variables will be accessible to the error handling action via request . va rs 
and can be used in generating the error response. In particular, it is a good 
idea for the error action to return the original HTTP error code instead of the 
default 200 (OK) status code. This can be done by setting response, status = 
request .vars. code. It is also possible to have the error action send (or queue) 
an email to an administrator, including a link to the ticket in admin. 

Unmatched errors display a default error page. This default error page can 
also be customized here (see router, example, py and routes, example, py in the 
root web2py folder): 

] error_message = ' <htmlxbodyxhl>%s</hlx/bodyx/html>' 

z error_message_ticket = ' ' ' <htmlxbodyxhl>Internal error</hl> 

3 Ticket issued: <a href =" /admin/ default/ ticket/%(ticket)s" 

4 target="^blank">%(ticket)s</ax/bodyx/html>' ' ' 

The first variable contains the error message when an invalid application or 
function is requested. The second variable contains the error message when 
a ticket is issued. 

routes_onerror work with both routing mechanisms 


4.18 Running tasks in the background 

In web2py, every http request is served in its own thread. Threads are 
recycled for efficiency and managed by the web server. For security, the 
web server sets a time-out on each request. This means that actions should 
not run tasks that take too long, should not create new threads, and should 
not fork processes (it is possible but not recommended). 
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The proper way to run time-consuming tasks is doing it in the background. 
There is not a single was of doing it, but here we describe three mechanisms 
that are built into web2py: cron, homemade task queues, and scheduler. 

By cron we refer to a web2py functionality not to the Unix Cron mechanism. 
The web2py cron works on windows too. web2py cron is the way to go if 
you need tasks in the background at scheduled times and these tasks take a 
relatively short time compared to the time interval between two calls. Each 
task runs in its own process, and multiple tasks can run concurrently, but 
you have no control over how many tasks run. If accidentally one task 
overlaps with itself, it can cause a database lock and a spike in memory 
usage. web2py scheduler takes a different approach. The number of running 
processes is fixed, and they can run on different machines. Each process 
is called a worker. Each worker picks a task when available and executes 
it as soon as possible after the time when it is scheduled to run, but not 
necessarily at that exact time. There cannot be more processes running than 
the number of scheduled tasks and therefore no memory spikes. Scheduler 
tasks can be defined in models and are stored in the database. The web2py 
scheduler does not implement a distributed queue since it assumes that the 
time to distribute tasks is negligible compared with the time to run the tasks. 
Workers pick up the task from the database. 

Homemade tasks queues can be a simpler alternative to the web2py 
scheduler in some cases. 


4..18.1 Cron 

The web2py cron provides the ability for applications to execute tasks at 
preset times, in a platform-independent manner. 

For each application, cron functionality is defined by a crontab file: 
app/cron/crontab 

It follows the syntax defined in ref. [45] (with some extensions that are 
specific to web2py). 

This means that every application can have a separate cron configuration and 
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that cron config can be changed from within web2py without affecting the 
host OS itself. 

Here is an example: 

1 0-59/1 * * * * root python /path/to/python/script .py 

2 30 3 * * * root *applications/admin/cron/db_ vacuum. py 

3 */30 * * * * root **applications/admin/cron/something. py 

4 @reboot root *mycontroller/myfunction 

5 @hourly root *applications/admin/cron/expire_sessions. py 

The last two lines in this example use extensions to regular cron syntax to 
provide additional web2py functionality. 

The file "applications/admin/cron/expire_sessions.py" actually exists and ships 
with the admin app. It checks for expired sessions and deletes them. 
' ' applications 1 'admin/cron/crontab" runs this task hourly. 

If the task/script is prefixed with an asterisk (*) and ends with .py, it will 
be executed in the web2py environment. This means you will have all the 
controllers and models at your disposal. If you use two asterisks (**), the 
MODELS will not be executed. This is the recommended way of calling, as it 
has less overhead and avoids potential locking problems. 

Notice that scripts /functions executed in the web2py environment require 
a manual db. commit ( ) at the end of the function or the transaction will be 
reverted. web2py does not generate tickets or meaningful tracebacks in shell 
mode, which is how cron is run, so make sure that your web2py code runs 
without errors before you set it up as a cron task as you will likely not be able 
to see those errors when run from cron. Moreover, be careful how you use 
models: while the execution happens in a separate process, database locks 
have to be taken into account in order to avoid pages waiting for cron tasks 
that may be blocking the database. Use the ** syntax if you don't need to use 
the database in your cron task. 

You can also call a controller function, in which case there is no need to 
specify a path. The controller and function will be that of the invoking 
application. Take special care about the caveats listed above. Example: 

, */30 * * * * root ■unycontroller/myfunction 
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If you specify @ reboot in the first field in the crontab file, the given task will be 
executed only once, at web2py startup. You can use this feature if you want 
to pre-cache, check, or initialize data for an application on web2py startup. 
Note that cron tasks are executed in parallel with the application — if the 
application is not ready to serve requests until the cron task is finished, you 
should implement checks to reflect this. Example: 
(areboot * * * * root *mycontroll_er/myf unction 

Depending on how you are invoking web2py, there are four modes of 
operation for web2py cron. 

• soft cron: available under all execution modes 

• hard cron: available if using the built-in web server (either directly or via 
Apache mod_proxy) 

• external cron: available if you have access to the system's own cron service 

• No cron 

The default is hard cron if you are using the built-in web server; in all other 
cases, the default is soft cron. Soft cron is the default method if you are using 
CGI, FASTCGI or WSGI (but note that soft cron is not enabled by default in 
the standard wsgihandler. py file provided with web2py). 

Your tasks will be executed on the first call (page load) to web2py after the 
time specified in crontab; but only after processing the page, so no delay 
will be observed by the user. Obviously, there is some uncertainty regarding 
precisely when the task will be executed, depending on the traffic the site 
receives. Also, the cron task may get interrupted if the web server has a page 
load timeout set. If these limitations are not acceptable, see external cron. 
Soft cron is a reasonable last resort, but if your web server allows other cron 
methods, they should be preferred over soft cron. 

Hard cron is the default if you are using the built-in web server (either 
directly or via Apache mod_proxy). Hard cron is executed in a parallel 
thread, so unlike soft cron, there are no limitations with regard to run time 
or execution time precision. 

External cron is not default in any scenario, but requires you to have access 
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to the system cron facilities. It runs in a parallel process, so none of the 
limitations of soft cron apply. This is the recommended way of using cron 
under WSGI or FASTCGI. 

Example of line to add to the system crontab, (usually /etc/crontab): 

0-59/1 * * * * web2py cd /var/www/web2py/ && python web2py.py -J -C -D 1 » /tmp/ 
cron. output 2>&1 


If you are running external cron, make sure you add the -N command line 
parameter to your web2py startup script or config so there is no collision of 
multiple types of cron. Also, with external cron, make sure to add either -J 
(or -cron job, which is the same) as indicated above so that web2py knows 
that task is executed by cron. Web2py sets this internally with soft and hard 
cron. 

In cases where you do not need any cron functionality within a particular 
process, you can use the -N command line parameter to disable it. Note that 
this might disable some maintenance tasks (like the automatic cleaning of 
session folders). The most common use of this function is when you: 

• have already set up external cron triggered from the system (most 
common with WSGI setup). 

• want to debug your application without cron interfering either with 
actions or with output. 


4..18.2 Homemade task queues 

While cron is useful to run tasks at regular time intervals, it is not always the 
best solution to run a background task. For this purpose web2py provides 
the ability to run any python script as if it were inside a controller: 

python web2py.py -S app -M -N -R applications/app/private/myscript .py -A a b c 

where -S app tells web2py to run "myscript.py" as "app", -M tells web2py to 
execute models, -N tells web2py not to run cron, and -A a b c passes optional 
command line arguments sys . a rgs= [ ' a ' , ' b ' , ' c ' ] to "myscript.py". 
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This type of background process should not be executed via cron (except 
perhaps for cron ©reboot) because you need to be sure that no more than 
one instance is running at the same time. With cron it is possible that a 
process starts at cron iteration 1 and is not completed by cron iteration 2, so 
cron starts it again, and again, and again - thus jamming the mail server. 

In chapter 8, we will provide an example of how to use the above method to 
send emails. 


4.18.3 Scheduler (experimental) 

The web2py scheduler works very much like the task queue described in the 
previous sub-section with some differences: 

• It provides a standard mechanism for creating and scheduling tasks. 

• There is not a single background process but a set of workers processes. 

• The job of worker nodes can be monitored because their state, as well as 
the state of the tasks, is stored in the database. 

• It works without web2py but that is not documented here. 

The scheduler does not use cron, although one can use cron ©reboot to start 
the worker nodes. 

In the scheduler, a task is simply a function defined in a model (or in a 

module and imported by a model). For example: 

def task_add(a,b): 
return a+b 

Tasks will always be called in the same environment seen by controllers 
and therefore they see all the global variables defined in models, including 
database connections (db). Tasks differ from a controller action because 
they are not associated with an HTTP request and therefore there is no 
request .env. 

Once the tasks are defined, you need to enable the scheduler by adding the 
following code to your model: 

myscheduler = Scheduler(db, diet (task_add=task_add) ) 
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The first argument of the Scheduler class must be the database to be used by 
the scheduler to communicate with the workers. This can be the db of the app 
or another dedicated db, perhaps one shared by multiple apps. The Scheduler 
creates the tables it needs. The second argument is a Python dictionary of 
key :value pairs where key is the name you want to use to expose a task, and 
value is the actual name of the function that defines the task. 

Once the tasks are defined and the Scheduler is instantiated, all that is needed 
to do is to start the workers: 

python web2py.py -K myapp 

The - K option starts a worker. The argument of the - K option is a list of app 
names, separated by commas. These are the apps that will be served by this 
worker. One can start many workers. 

Now we have the infrastructure in place: defined the tasks, told the scheduler 
about them, started the worker(s). What remains is to actually schedule the 
tasks: 

Tasks can be scheduled programmatically or via appadmin. In fact, a task 
is scheduled simply by adding an entry in the table "scheduler_task", which 
you can access via appadmin: 

http://127.0.0.1:8000/scheduler/appadmin/insert/db/scheduler_task 

The meaning of the fields in this table is obvious. The "args" and "vars"" 
fields are the values to be passed to the task in JSON format. In the case of 
the "task_add" above, an example of "args" and "vars" could be: 

args = [3, 4] 
vars = {} 

or 

args = [] 

vars = { 'a ' :3 : ' b : :4} 

A task can be in one of the following states: 

QUEUED, RUNNING, COMPLETED, FAILED, TIMEOUT 
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Once a task exists (there is a record in the "scheduler_task" table), is QUEUED, 
and is ready (meets all the conditions specified in the record), it can be 
picked up by a worker. As soon as a worker is available, it picks the first 
ready task scheduled to run. The worker creates an entry in a another table, 
"scheduler_run" (also created by the scheduler). 

The table "scheduler_run" stores the status of all running tasks. Each record 
references a task that has been picked up by a worker. One task can have 
multiple runs. For example, a task scheduled to repeat 10 times an hour will 
probably have 10 runs (unless one fails or they take longer than 1 hour). 

Possible run statuses are: 

RUNNING, COMPLETED, FAILED, TIMEOUT 

When a QUEUED task is picked up, it becomes a RUNNING task and its run status 
is also RUNNING. If the run is completed, no exceptions are thrown, and there 
is no task timeout, the run is marked as COMPLETED and the task is marked as 
QUEUED or COMPLETED depending on whether it is supposed to run again at a 
later time. The output of the task is serialized in JSON and stored in the run 
record. 

When a RUNNING task throws an exception, the run is mark as FAILED and the 
task is marked as FAILED. The traceback is stored in the run record. 

Similarly, when a run exceeds the timeout, it is stopped and marked as 
TIMEOUT, and the task is marked as TIMEOUT. 

In any case, the stdout is captured and also logged into the run record. 

Using appadmin, one can check all RUNNING tasks, the output of COMPLETED 
tasks, the error of FAILED tasks, etc. 

The scheduler also creates one more table called "scheduler_worker", which 
stores the workers' heartbeat and their status. Possible worker statuses are: 

ACTIVE, INACTIVE, DISABLED 

You can disable a worker by changing its status using appadmin. 

Everything that one can do via appadmin one can do programmatically by 
inserting and updating records in these tables. 
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Anyway, one should not update records relative to RUNNING tasks as this may 
create an un-expected behavior. The best practice is to queue tasks using 
"insert". For example: 

db. schedule r_t as k. insert ( 
status=' QUEUED' , 
application_name= ' myapp' , 
task_name='my first task', 
function_name=' task_add' , 
args='f7' , 

vars="{'a' :3, ' b' :4}" , 
enabled=True, 
start_time = request. now, 

stop_time = request .now+datetime.timedelta(days=l) , 
repeats = 10, # run 18 times 
period = 3600, # every lh 
timeout = 60, # should take less than 60 seconds 


Notice that fields "times_run", "last_run_time" and "assigned_worker_name" 
are not provided at schedule time but are filled automatically by the workers. 

You can also retrieve the output of completed tasks: 

completed- runs = db (db. scheduler. run. status^' COMPLETED ' ) .select ( ) 

The scheduler is experimental because it needs more extensive testing and 
because the table structure may change as more features are added. 

• We recommend having a separate model file for definindg the tasks and 
instantiate Scheduler (after te tasks are defined). 

• We recommend that use at least one worker per application so that you 
have more control, although that is not strictly necessary. 

• If your tasks are defined in a module (as opposed to a model) you may 
have to restart the workers. 


4.29 Third party modules 

web2py is written in Python, so it can import and use any Python module, 
including third party modules. It just needs to be able to find them. As with 
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any Python application, modules can be installed in the official Python "site- 
packages" directory, and they can then be imported from anywhere inside 
your code. 

Modules in the "site-packages" directory are, as the name suggests, site-level 
packages. Applications requiring site-packages are not portable unless these 
modules are installed separately. The advantage of having modules in "site- 
packages" is that multiple applications can share them. Let's consider, for 
example, the plotting package called "matplotlib". You can install it from the 
shell using the PEAK easy_install command: 

easy_install py-matplotlib 

and then you can import it into any model/controller/view with: 
import matplotlib 

The web2py source distribution, and the Windows binary distribution has 
a site-packages in the top-level folder. The Mac binary distribution has a 
site-packages folder in the folder: 

web2py .app/Contents/Resources/ site -packages 

The problem with using site-packages is that it becomes difficult to use 
different versions of a single module at the same time, for example there 
could be two applications but each one uses a different version of the same 
file. In this example, sys . path cannot be altered because it would affect both 
applications. 

For this kind of situation, web2py provides another way to import modules 
in such a way that the global sys . path is not altered: by placing them in the 
"modules" folder of an application. One side benefit is that the module will 
be automatically copied and distributed with the application. 

Once a module "mymodule.py" is placed into an app "modules/" folder, it can 
be imported from anywhere inside a webzpy application (without need to alter 
sys. path with): 

1 import mymodule 
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4.20 Execution environment 

While everything discussed here works fine, we recommend instead building 
your application using components, as described in chapter 12. 

web2py model and controller files are not Python modules in that they 
cannot be imported using the Python import statement. The reason for 
this is that models and controllers are designed to be executed in a 
prepared environment that has been pre-populated with web2py global 
objects (request, response, session, cache and T) and helper functions. This is 
necessary because Python is a statically (lexically) scoped language, whereas 
the web2py environment is created dynamically. web2py provides the 
exec_environment function to allow you to access models and controllers 
directly. exec_environment creates a web2py execution environment, loads 
the file into it and then returns a Storage object containing the environment. 
The Storage object also serves as a namespace mechanism. Any Python file 
designed to be executed in the execution environment can be loaded using 
exec_environment. Uses for exec_environment include: 

• Accessing data (models) from other applications. 

• Accessing global objects from other models or controllers. 

• Executing controller functions from other controllers. 

• Loading site-wide helper libraries. 

This example reads rows from the user table in the cas application: 

1 from gluon. shell import exec_environment 

2 cas = exec_environment( 'applications/ cas/models/db.py ' ) 

3 rows = cas .db( ). select(cas .db. user. ALL) 

Another example: suppose you have a controller "other.py" that contains: 

1 def some_action( ) : 

2 return diet ( remote_addr=request.env. remote_addr) 

Here is how you can call this action from another controller (or from the 
web2py shell): 
1 from gluon. shell import exec_environment 
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other = exec_environment( ' applications/app/controllers/other.py ' , request=request) 
result = other. some_action( ) 

In line 2, request= request is optional. It has the effect of passing the 
current request to the environment of "other". Without this argument, the 
environment would contain a new and empty (apart from request .folder) 
request object. It is also possible to pass a response and a session object 
to exec_environment. Be careful when passing request, response and session 
objects — modification by the called action or coding dependencies in the 
called action could lead to unexpected side effects. 

The function call in line 3 does not execute the view; it simply returns the 
dictionary unless response, render is called explicitly by "some_action". 

One final caution: don't use exec_environment inappropriately. If you want 
the results of actions in another application, you probably should implement 
an XML-RPC API (implementing an XML-RPC API with web2py is almost 
trivial). Don't use exec_environment as a redirection mechanism; use the 
redirect helper. 

4.22 Cooperation 

There are many ways applications can cooperate: 

• Applications can connect to the same database and thus share tables. It is 
not necessary that all tables in the database are defined by all applications, 
but they must be defined by those applications that use them. All 
applications that use the same table, but one, must define the table with 
migrate=False. 

• Applications can embed components from other applications using the 
LOAD helper (described in Chapter 12). 

• Applications can share sessions. 

• Applications can call each other's actions remotely via XML-RPC. 

• Applications can access each other 's files via the filesystem (assuming they 
share the same filesystem). 
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• Applications can call each other's actions locally using exec_environment as 
discussed above. 

• Applications can import each other's modules using the syntax: 
from applications. appname. modules import mymodule 

• Applications can import any module in the PYTHONPATH search path, 
sys. path. 

One app can load the session of another app using the command: 
session. connect (request, response, masterapp= ' appname ' , db=db) 

Here "appname" is the name of the master application, the one that sets the 
initial session_id in the cookie, db is a database connection to the database 
that contains the session table (web2py_session). All apps that share sessions 
must use the same database for session storage. 

One application can load a module from another app using 

import applicat ion s.ot he rapp. modules. ot he rmodule 


4.22 Logging 

Python provides logging APIs. Web2py provides a mechanism to configure 
it so that apps can use it. 

In your application, you can create a logger, for example in a model: 

1 import logging 

2 logger = logging. getl_ogger( "web2py. app. myapp" ) 

3 logger. setLevel (logging. DEBUG) 

and you can use it to log messages of various importance 

1 logger. debug( "Just checking that %s" % details) 

2 logger. info('Tou ought to know that %s" % details) 
, logger. warn( "Mind that %s" % details) 

4 logger. error( "Oops, something bad happened %s" % details) 

logging is a standard python module described here: 
1 http://docs . python.org/library/logging.html 
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The string "web2py.app.myapp" defines an app-level logger. 

For this to work properly, you need a configuration file for the logger. One is 
provided by web2py in the root web2py folder "logging. example. conf". You 
need to rename the file "logging. conf" and customize it as necessary. 

This file is self documenting, so you should open it and read it. 

To create a configurable logger for application "myapp", you must add 
myapp to the [loggers] keys list: 

■ [loggers] 

2 keys=root, rocket, ma rkdown,web2py, rewrite, app, welcome, myapp 

and you must add a [logger_myapp] section, using [logger_welcome] as a 
starting point. 

1 [logger„myapp] 

2 level=WARNING 

3 qualname=web2py. app. myapp 

4 handlers=consoleHandler 

5 propagate=0 

The "handlers" directive specifies the type of logging and here it is logging 
"myapp" to the console. 


4.23 WSGI 

web2py and WSGI have a love-hate relationship. Our perspective is that 
WSGI was developed as a protocol to connect web servers to web applications 
in a portable way, and we use it for that purpose. web2py at its core is 
a WSGI application: gluon.main.wsgibase. Some developers have pushed 
WSGI to its limits as a protocol for middleware communications and develop 
web applications as an onion with many layers (each layer being a WSGI 
middleware developed independently of the entire framework). web2py 
does not adopt this structure internally. This is because we feel the core 
functionality of a frameworks (handling cookies, session, errors, transactions, 
dispatching) can be better optimized for speed and security if they are 
handled by a single comprehensive layer. 
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Yet web2py allows you to use third party WSGI applications and middleware 
in three ways (and their combinations): 

• You can edit the file "wsgihandler.py" and include any third party WSGI 
middleware. 

• You can connect third party WSGI middleware to any specific action in 
your apps. 

• You can call a third party WSGI app from your actions. 

The only limitation is that you cannot use third party middleware to replace 
core web2py functions. 

4..23.1 External middleware 
Consider the file "wsgibasepy": 



When LOGGING is set to True, gluon.main.wsgibase is wrapped by the 
middleware function gluon.main.appfactory. It provides logging to the 
"httpserver.log" file. In a similar fashion you can add any third party 
middleware. We refer to the official WSGI documentation for more details. 


4.23.2 Internal middleware 

Given any action in your controllers (for example index) and any third party 
middleware application (for example MyMiddleware, which converts output to 
upper case), you can use a web2py decorator to apply the middleware to that 
action. Here is an example: 

class MyMiddleware: 
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2 

converts output to upper case 

3 

def init (self,app): 

4 

self.app = app 

5 

def call (self, environ, start_response) : 

6 

items = self .app(environ, start_response) 

7 

return [item. upper ( ) for item in items] 

9 

@ request . wsgi . middlewa re (MyMiddlewa re ) 

[0 

def index( ) : 

" 

return 'hello world' 


We cannot promise that all third party middleware will work with this 
mechanism. 


4.23.3 Calling WSGI applications 

It is easy to call WSGI app from a web2py action. Here is an example: 

def test_wsgi_app(environ, start. response) : 

this is a test WSGI app 

status = '200 OK' 

response_headers = [(' Content-type' ,' text/ plain' ) , 

('Content -Length' , '13')] 
start- response (stat us, response_headers) 
return ['hello world! \n' ] 

def index( ) : 

a test action that calls the previous app and escapes output 

items = test_wsgi_app (request. wsgi. environ, 

request .wsgi. start- response) 
for item in items: 

response. write (it em, escape=False) 
return response. body. getvalue( ) 

In this case, the index action calls test_wsgi_app and escapes the returned 
value before returning it. Notice that index is not itself a WSGI app and 
it must use the normal web2py API (such as response. write to write to the 
socket). 


5 

The views 


web2py uses Python for its models, controllers, and views, although it uses 
a slightly modified Python syntax in the views to allow more readable code 
without imposing any restrictions on proper Python usage. 

The purpose of a view is to embed code (Python) in an HTML document. In 
general, this poses some problems: 

• How should embedded code be escaped? 

• Should indenting be based on Python or HTML rules? 

web2py uses {{ ... }} to escape Python code embedded in HTML. The 
advantage of using curly brackets instead of angle brackets is that it's 
transparent to all common HTML editors. This allows the developer to use 
those editors to create web2py views. 

Since the developer is embedding Python code into HTML, the document 
should be indented according to HTML rules, and not Python rules. 
Therefore, we allow unindented Python inside the {{ ... }} tags. Since 
Python normally uses indentation to delimit blocks of code, we need a 
different way to delimit them; this is why the web2py template language 
makes use of the Python keyword pass. 

A code block starts with a line ending with a colon and ends with a line 
beginning with pass. The keyword pass is not necessary when the end of 
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the block is obvious from the context. 
Here is an example: 



Note that pass is a Python keyword, not a web2py keyword. Some Python 
editors, such as Emacs, use the keyword pass to signify the division of blocks 
and use it to re-indent code automatically. 

The web2py template language does exactly the same. When it finds 
something like: 

1 <htmlxbody> 

2 {{for x in range(10) :}}{{=x}}hello<br />{{pass}} 

3 </bodyx/html> 

it translates it into a program: 

1 response. write( <htmlxbody> , escape=False) 

2 for x in range(10) : 

3 response. write(x) 

4 response. write( hello<br /> , escape=False) 

5 response. write( </bodyx/html> , escape=False) 

response. write writes to the response, body. 

When there is an error in a web2py view, the error report shows the generated 
view code, not the actual view as written by the developer. This helps the 
developer debug the code by highlighting the actual code that is executed 
(which is something that can be debugged with an HTML editor or the DOM 
inspector of the browser). 

Also note that: 

■ {{=X}} 

generates 
] response. write(x) 
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Variables injected into the HTML in this way are escaped by default. The 
escaping is ignored if x is an XML object, even if escape is set to True. 

Here is an example that introduces the HI helper: 

{{=Hl(i)}} 

which is translated to: 
response. write(Hl(i) ) 

upon evaluation, the HI object and its components are recursively serialized, 
escaped and written to the response body. The tags generated by HI and 
inner HTML are not escaped. This mechanism guarantees that all text — and 
only text — displayed on the web page is always escaped, thus preventing 
XSS vulnerabilities. At the same time, the code is simple and easy to debug. 

The method response. write(obj , escape=True) takes two arguments, the 
object to be written and whether it has to be escaped (set to True by default). 
If ob j has an . xml ( ) method, it is called and the result written to the response 

body (the escape argument is ignored). Otherwise it uses the object's str 

method to serialize it and, if the escape argument is True, escapes it. All built- 
in helper objects (HI in the example) are objects that know how to serialize 
themselves via the . xml ( ) method. 

This is all done transparently. You never need to (and never should) call the 
response. write method explicitly. 


5.2 Basic syntax 

The web2py template language supports all Python control structures. Here 
we provide some examples of each of them. They can be nested according to 
usual programming practice. 


5.1.1 for... in 

In templates you can loop over any iterable object: 
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which produces: 

1 <ul> 

2 <li>a</li> 

3 <li>b</li> 

4 <li>c</li> 

5 </ul> 

Here item is any iterable object such as a Python list, Python tuple, or 
Rows object, or any object that is implemented as an iterator. The elements 
displayed are first serialized and escaped. 

5.I.2 while 

You can create a loop using the while keyword: 


{{k = 3}} 


<ul> 


{{while k > 0:}}<li>{{=k}}{{k = k 

- 1}}</U>{{pass}} 

</ul> 



which produces: 

<ul> 

<U>3</U> 

<li>2</li> 

<li>l</li> 

</ul> 


5.1.3 if . . .elif . . .else 


You can use conditional clauses: 

. {{ 

z import random 

3 k = random. randint(0, 100) 

4 }} 

, <h2> 
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6 {{=k}} 

7 {{if k % 2:}}is odd{{else: }}is even{{pass}} 
s </h2> 


which produces: 

, <h2> 

z 45 is odd 

3 </h2> 

Since it is obvious that else closes the first if block, there is no need for a pass 
statement, and using one would be incorrect. However, you must explicitly 
close the else block with a pass. 

Recall that in Python "else if" is written elif as in the following example: 

. {{ 

z import random 

, k = random. randint(0, 100) 

4 }} 

5 <h2> 

6 {{=k}} 

- {{if k % 4 == 0:}}is divisible by 4 

s {{elif k % 2 == 0:}}is even 

9 {{else:}}is odd 

.0 {{pass}} 

„ </h2> 

It produces: 

, <h2> 

2 64 is divisible by 4 

3 </h2> 


5.I.4 try. . .except. . .else. . .finally 

It is also possible to use try. . .except statements in views with one caveat. 
Consider the following example: 

, {{try:}} 

2 Hello {{= 1 / 0}} 

3 {{except:}} 

4 division by zero 

5 {{else:}} 

6 no division by zero 
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7 {{finally}} 
s <br /> 
9 {{pass}} 

It will produce the following output: 

, Hello 

2 division by zero 

3 <br /> 


This example illustrates that all output generated before an exception occurs 
is rendered (including output that preceded the exception) inside the try 
block. "Hello" is written because it precedes the exception. 


5.I.5 def . . . return 

The web2py template language allows the developer to define and implement 
functions that can return any Python object or a text/html string. Here we 
consider two examples: 



produces the following output: 

1 <ul> 

2 <lixa href="/7ttp;/Mt«. google. cora">www. google. com</ax/li> 

3 </ul> 

The function itemizel returns a helper object that is inserted at the location 
where the function is called. 

Consider now the following code: 
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It produces exactly the same output as above. In this case, the function 
itemize2 represents a piece of HTML that is going to replace the web2py 
tag where the function is called. Notice that there is no '=' in front of the call 
to itemize2, since the function does not return the text, but it writes it directly 
into the response. 

There is one caveat: functions defined inside a view must terminate with a 
return statement, or the automatic indentation will fail. 

5.2 HTML helpers 

Consider the following code in a view: 

{{=DIV( 'this', 'is', 'a', 'tesf, _id='I23', _class='mycZass' )}} 

it is rendered as: 

<div id="I23" class="myclass">thisisatest</div> 

DIV is a helper class, i.e., something that can be used to build HTML 
programmatically. It corresponds to the HTML <div> tag. 

Positional arguments are interpreted as objects contained between the 
open and close tags. Named arguments that start with an underscore 
are interpreted as HTML tag attributes (without the underscore). Some 
helpers also have named arguments that do not start with underscore; these 
arguments are tag-specific. 

Instead of a set of unnamed arguments, a helper can also take a single list or 
tuple as its set of components using the * notation and it can take a single 
dictionary as its set of attributes using the **, for example: 



(produces the same output as before). 
The following set of helpers: 
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A, B, BEAUTIFY, BODY, BR, CAT, CENTER, CODE, COL, COLGROUP, DIV, EM, EMBED, FIELDSET, 
FORM, HI, H2, H3, H4, H5, H6, HEAD, HR, HTML, I, IFRAME, IMG, INPUT, LABEL, LEGEND, LI, 
LINK, MARKMIN, MENU, META, OBJECT, ON, OL, OPTGROUP, OPTION, P, PRE, SCRIPT, SELECT, 
SPAN, STYLE, TABLE, TAG, TBODY, TD, TEXTAREA, TFOOT, TH, THEAD, TITLE, TR, TT, UL, URL, 
XHTML, XML, embed64, xmlescape can be used to build complex expressions that 
can then be serialized to XML [51] [52]. For example: 
, {{=DIV(B(I("heZlo ", "<world>") ) ) , _class="myclass" )}} 

is rendered: 

1 <div class="myclass"xbxi>hello &lt;world&gt ;</ix/t»</div> 

Helpers can also be serialized into strings, equivalently, with the str and 

the xml methods: 

, >» print str(DIV("hello world")) 

z <div>hello world</div> 

3 >» print DIV("/?eIlo world") .xml( ) 

4 <div>hello world</div> 

The helpers mechanism in web2py is more than a system to generate HTML 
without concatenating strings. It provides a server-side representation of the 
Document Object Model (DOM). 

Components of helpers can be referenced via their position, and helpers act 
as lists with respect to their components: 

, >» a = DIV(SPAN('a' , 'b') , '<:') 

2 >» print a 

3 <divxspan>ab</span>c</div> 

4 >» del a[l] 

5 >» a.append(B( 'x' ) ) 
1, >» a[0] [0] = 'y' 

- >» print a 

s <divxspan>yb</spanxb>x</bx/div> 

Attributes of helpers can be referenced by name, and helpers act as 
dictionaries with respect to their attributes: 


>» a = DIV(SPAN('a' , 'b') , 

•C) 

>» a[ ' class' ] = ' s ' 


>» a[0] [ ' class' ] = ' t' 


>» print a 


<div class="s"xspan class= 

"t">ab</span>c</div> 
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Note, the complete set of components can be accessed via a list called 
a. components, and the complete set of attributes can be accessed via a 
dictionary called a. attributes. So,a[i] is equivalent to a. components [i] when 
i is an integer, and a[s] is equivalent to a.attributes[s] when s is a string. 

Notice that helper attributes are passed as keyword arguments to the helper. 
In some cases, however, attribute names include special characters that are 
not allowed in Python identifiers (e.g., hyphens) and therefore cannot be used 
as keyword argument names. For example: 

■ DIV('text', -data- role-' collapsible') 

will not work because "_data-role" includes a hyphen, which will produce a 
Python syntax error. 

In such cases, you can instead pass the attributes as a dictionary and make 
use of Python's ** function arguments notation, which map a dictionary of 
(key:value) pairs into a set of keyword arguments: 

i >» print DIV('texr', **{' ^data-role' : 'collapsible'}) 
z <div data- ro\e="collapsible">text</di\i> 

You can also dynamically create special TAGs: 

i >» print TAG[ ' soap:Body' ]( 'whatever' ,**{ '_xmZns:/n' : ' http:/ /www. example, org' }) 
2 <soap:Body xmlns :m=" http :// www . example . org ">whatever</soap:Body> 


5.2.1 XML 

XML is an object used to encapsulate text that should not be escaped. The text 
may or may not contain valid XML. For example, it could contain JavaScript. 

The text in this example is escaped: 

>» print DI\l(."<b>hello</b>") 
&lt ; b&gt ; hello&lt ; /b&gt ; 


by using XML you can prevent escaping: 

■ >» print DIV(XML("<i)>heno</b>")) 
2 <b>hello</b> 
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Sometimes you want to render HTML stored in a variable, but the HTML 
may contain unsafe tags such as scripts: 

>» print XML['<script>alert("unsafe!" )</script>' ) 
<script>alert ( "unsafe! ")</script> 

Un-escaped executable input such as this (for example, entered in the body 
of a comment in a blog) is unsafe, because it can be used to generate Cross 
Site Scripting (XSS) attacks against other visitors to the page. 

The web2py XML helper can sanitize our text to prevent injections and escape 
all tags except those that you explicitly allow. Here is an example: 

>» print XHL(' <script>alert("unsafe!" )</script>' , sanitize=True) 
&lt ; script&gt ; alert (&quot ; unsafe ! &quot ; )&lt ; /script&gt ; 

The XML constructors, by default, consider the content of some tags and some 
of their attributes safe. You can override the defaults using the optional 
permittecLtags and allowecLattributes arguments. Here are the default 
values of the optional arguments of the XML helper. 



5.2.2 Built-in helpers 

A 

This helper is used to build links. 

>» print h('<click>' , XHL( '<b>me</b>' ) , 

_href=' http:/ /www. web2py. com' ) 
<a h ref = 'http: //www. web2py. com ' >&lt ; click&gt ; <t»me/t»</a> 

Instead of _href you can pass the URL using the callback argument. For 
example in a view: 

{{=A( ' click me' , callback=URL( 'myaction ' ) )}} 
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and the effect of pressing the link will be an ajax call to "myaction" instead of 
a redirection. In this case, optionally you can specify two more arguments: 
target and delete: 

1 {{=A( 'click me' , caUback=URI_( 'myaction' ) , target="t" )}} 
z <div id="t"xdiv> 

and the response of the ajax callback will be stored in the DIV with id equal 
to "t". 

1 <div id="6">{{=A( ' click me' , callback=URL( 'myaction' ) , delete=' div#b" )}}</div> 

and upon response, the closest tag matching "div#b" will be deleted. In this 
case, the button will be deleted. A typical application is: 

■ {{=A( 'click me' , caUback=URL ( 'myaction ') , delete=' tr")}} 

in a table. Pressing the button will perform the callback and delete the table 
row. 

callback and delete can be combined. 

Tha A helper takes a special argument called cid. It works as follows: 

1 {{=A( ' linked page' , _href=' http://example.com' , cid='myic(' )}} 

2 <div id="myicf"></div> 

and a click on the link causes the content to be loaded in the div. This 
is similar but more powerful than the above syntax since it is designed to 
refresh page components. We discuss applications of cid in more detail in 
Chapter 12, in the context of components. 

These ajax features require jQuery and "static/js/web2py_ajax.js", which 
are automatically included by placing {{include 'web2py_ajax.html'}} in the 
layout head, "views/ web2py_ajax.html" defines some variables based on 
request and includes all necessary js and ess files. 


This helper makes its contents bold. 

1 >» print B( '<hello>' , XML ( '<i>world</i>' ) , _class=' test' , _id=0) 

2 <b id="0" class="test">&lt ;hello&gt;<i>world</ix/b> 
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BODY 

This helper makes the body of a page. 

>» print BODY (,' <hello> ' , XMLC <b>world</b>' ) , _bgcolor=' reef' ) 
<body bgcolo r=" red" >&lt ; hello&gt ; <b>wo rld</bx/body> 


BR 


This helper creates a line break. 


>» print BR() 
<br /> 


CAT (1.98.1 and up) 

This helper concatenates other helpers, same as TAG["]. 


>» 

print 

CAT ('Here is a ' , M'link' ,. 

.href=URL()), ', 

and 

he, 

^e is 

some ' , 

B( 

f bold 


text- 

), '. ') 









Here 

is a 

<a href=' 

' /app/ default/ index' 

'>link</a>, and 

here 

is 

some 

<b>bold 

text</b>. 


CENTER 

This helper centers its content. 

, >» print CENTER('<heno>' , XHL( ' <b>world</b>' ) , 

2 >» _class=' test ' , _id=0) 

3 <center id="0" class="test">&lt ;hello&gt ;<b>world</bx/center> 

CODE 

This helper performs syntax highlighting for Python, C, C++, HTML and 
web2py code, and is preferable to PRE for code listings. CODE also has the 
ability to create links to the web2py API documentation. 

Here is an example of highlighting sections of Python code. 

1 >» print C0DE( 'print "hello"', language= 'python ') .xml() 

2 <tablextr valign="top"xtd style='Vidth:40px; text-align: right;"xpre style=" 

3 font-size: llpx; 

4 font- family: Bitstream Vera Sans Mono, monospace; 

5 background-color: transparent; 

6 margin: 0; 

7 padding: 5px; 

8 border: none; 

i) background-color: #EQEQEB; 

io color: #A0A0AO; 
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u ">l.</pre></tdxtdxpre style=" 

12 font-size: llpx; 

13 font- family: Bitstream Vera Sans Mono, monospace; 

14 background- color: transparent; 

15 margin: 0; 

16 padding: 5px; 
i 7 border: none; 

18 overflow: auto; 

19 "xspan style=" color: #185369; font-weight: bold">print </span> 

20 <span style="color: #FF9966">"hetto"</span></pre></td></tr> 
2, </table> 

Here is a similar example for HTML 

■ >» print C0DE( 

2 >» ' <htmlxbody>{{=request.env. remote^add}}</bodyx/html>' , 

3 >» language^ ' html ' ) 

1 <table>. . .<code>. . . 

2 <htmlxbody>{{=request . env . remote_add}}</bodyx/html> 
, . . .</code>. . .</table> 

These are the default arguments for the CODE helper: 
] CODE("print 'hello world'", language^ ' python ' , link=None, counter=l, styles={}) 

Supported values for the language argument are "python", "html_plain", "c", 
"cpp", "web2py", and "html". The "html" language interprets {( and }} tags as 
"web2py" code, while "html_plain" doesn't. 

If a link value is specified, for example "/examples/global/vars/", web2py 
API references in the code are linked to documentation at the link URL. 
For example "request" would be linked to "/examples/global/vars/request". 
In the above example, the link URL is handled by the "vars" action in the 
"global. py" controller that is distributed as part of the web2py "examples" 
application. 

The counter argument is used for line numbering. It can be set to any of 
three different values. It can be None for no line numbers, a numerical value 
specifying the start number, or a string. If the counter is set to a string, it is 
interpreted as a prompt, and there are no line numbers. 

The styles argument is a bit tricky. If you look at the generated HTML 
above, it contains a table with two columns, and each column has its own 
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style declared inline using CSS. The styles attributes allows you to override 
those two CSS styles. For example: 
1 {{=C0DE( ... ,styles={ 'CODE' : 'margin: 0;padding: 5px;border: none;'})}} 

The styles attribute must be a dictionary, and it allows two possible keys: 
CODE for the style of the actual code, and LINENUMBERS for the style of the left 
column, which contains the line numbers. Mind that these styles completely 
replace the default styles and are not simply added to them. 

COL 

1 >» print C0L( ' a' ,' b' ) 
z <col>ab</col> 

COLGROUP 

, >» print COLGROUP ('a' ,'b') 

2 <colgroup>ab</colgroup> 

DIV 

All helpers apart from XML are derived from DIV and inherit its basic methods. 

1 >» print DIV( '<hello>' , XHL( ' <b>wo rld</ b> ' ) , _class=' test ' , _id=0) 

2 <div id="0" class="test">&lt ; hello&gt;<t»world</t»</div> 

EM 

Emphasizes its content. 

, >» print EH( '<hello>' , Xm.('<b>world</b>' ) , _class=' test' , _id=0) 
2 <em id="0" class="test">&lt ;hello&gt;<b>world</b></em> 

FIELDSET 

This is used to create an input field together with its label. 

, >» print FIELDSET (' Height: ', INPUT(_name=' height ' ) , _class=' test ' ) 
2 <fieldset class="test">Height :<input name="/7eig/7t" /></fieldset> 

FORM 

This is one of the most important helpers. In its simple form, it just makes a 
<form>. . .</form> tag, but because helpers are objects and have knowledge of 
what they contain, they can process submitted forms (for example, perform 
validation of the fields). This will be discussed in detail in Chapter 7. 
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1 >» print FORM(INPUT(_type= ' submit ') , _action='', _method=' post ' ) 

2 <form enctype="mulripart/ form-data" action="" method="post"> 

3 <input type="submit" /></form> 

The "enctype" is "multipart/form-data" by default. 

The constructor of a FORM, and of SQLFORM, can also take a special argument 
called . When a dictionary is passed as , its items are translated into "hidden" 
INPUT fields. For example: 

■ >» print FORM(hidden=dict(a='b' ) ) 

2 <form enctype="multipart/ form-data" action="" method="post"> 

3 <input value="fa" type="hidden" name="a" /></form> 

HI, H2, H3, H4, H5, H6 

These helpers are for paragraph headings and subheadings: 

, >» print Hl('</7elIo> 1 , XHL( ^<b>world</b>^ ) , _class=' test ' , _id=0) 
2 <hl id="0" class="test">&lt ;hello&gt;<b>world</bx/hl> 

HEAD 

For tagging the HEAD of an HTML page. 

, >» print HEAD(TrrLE( , </ieZlo> , 1 XHL( l <b>world</b> 1 ) ) ) 
2 <headxtitle>&lt;hello&gt ;<b>world</bx/titlex/head> 

HTML 

This helper is a little different. In addition to making the <html> tags, it 
prepends the tag with a doctype string [54, 55, 56]. 

, >» print HTML ( BODY (' <he 11 o>' , XHL( ' <b>uorld</b>' ) ) ) 

2 <!D0CTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional/ /EN" 

3 "http://www.w3. org/ TR/ html4/ loose . dtd"> 

4 <htmlxbody>&lt ;hello&gt;<b>world</b></bodyx/html> 

The HTML helper also takes some additional optional arguments that have 
the following default: 

i HTML( . . . , lang='en', doctype^' transitional ' ) 

where doctype can be 'strict', 'transitional', 'frameset', 'html^', or a full 
doctype string. 
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XHTML 

XHTML is similar to HTML but it creates an XHTML doctype instead. 
XHTML(..., lang='en', doctype=' transitional' , xmlns=' http://www.w3.org/1999/xhtml' ) 

where doctype can be 'strict', 'transitional', 'frameset', or a full doctype 
string. 

HR 

This helper creates a horizontal line in an HTML page 

>» print HR() 
<hr /> 


This helper makes its contents italic. 

1 >» print I( '<hello>' , XHL( '<b>world</b>' ) , _class=' test' , _id=0) 

2 <i id="0" class="rest">&lt ;hello&gt;<t»world</t»</i> 

INPUT 

Creates an <input. . ./> tag. An input tag may not contain other tags, and is 
closed by /> instead of >. The input tag has an optional attribute -type that 
can be set to "text" (the default), "submit", "checkbox", or "radio". 

1 >» print INPUT(_name=' test ' , _value='a') 
z <input value="a" name="test" /> 

It also takes an optional special argument called "value", distinct from 
"_value". The latter sets the default value for the input field; the former 
sets its current value. For an input of type "text", the former overrides the 
latter: 

1 >» print INPUT(_name=' test ' , _value='a', value='b') 

2 <input value="£>" name="test" /> 

For radio buttons, INPUT selectively sets the "checked" attribute: 

1 >» for v in [' a' , ' b' , ' c' ] : 

2 >» print INPUT (_type=' radio' , _name=' test ' , _value=v, value='b'), v 
, <input value="a" type="racfio" name="test" /> a 

4 <input value="fa" type="radio" checked="chec/<ecf" name="test" /> b 

5 <input value="c" type="radio" name="test" /> c 
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and similarly for checkboxes: 

1 >» print INPUT (_type=' checkbox' , _name=' test ' , _value='a', value=True) 

2 <input value="a" type=" checkbox" checked=" checked" name="test" /> 

3 >» print INPUT(_type=' checkbox' , _name=' test ' , _value='a', value=False) 

4 <input value="a" type=" checkbox" name="test" /> 

I FRAME 

This helper includes another web page in the current page. The url of the 
other page is specified via the "_src" attribute. 

1 >» print IFRtME{_src=' http://www.web2py.com' ) 

2 <iframe src="http://www. web2py. com"x/if rame> 

IMG 

It can be used to embed images into HTML: 

1 >» IV\G(src=' http: //example, com/image. png' ,_alt=' rest ' ) 

2 <img src="http: //example. com/ image. ong" alt="rest" /> 

Here is a combination of A, IMG, and URL helpers for including a static 
image with a link: 


1 

>» A(IMG(_src=URL( 'static' , ' logo. png' ) , 

_alt="My Logo") , 

2 

3 

_href=URL('cfefault' , 'index' )) 
<a href -' Imyapp/ default/ index" > 


4 

<img src=" /myappl static/ logo. png" alt= 

"My Logo" /> 

5 

</a> 



LABEL 


It is used to create a LABEL tag for an INPUT field. 

, >» print LABEL('</7eno>' , XHL( '<b>world</b>' ) , _class=' test ' , _id=0) 
2 <label id="0" class="test">&lt ;hello&gt;<b>world</bx/label> 


LEGEND 

It is used to create a legend tag for a field in a form. 

1 >» print LEGEND ( 'Name' , _for=' my field' ) 

2 <legend for="myfieIcf">Name</legend> 


LI 

It makes a list item and should be contained in a UL or OL tag. 
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>» print LI( '<hello>' , XML ( '<b>world</b>' ) , _class=' test ' , _id=0) 
<li id="0" class="test">&lt ;hello&gt;<t»world</t»</li> 


META 

To be used for building META tags in the HTML head. For example: 

1 >» print META(_name=' security ' , _content= ' high ' ) 

2 <meta name="security" content="/?igh" /> 

MARKMIN 

Implements the markmin wiki syntax. It converts the input text into output 
html according to the markmin rules described in the example below: 

1 >» print MARKMIN("this is **bold** or ''italic' 1 and this [[a link http://web2py. 

com]]") 

2 <p>this is <t»bold</b> or <i>italic</i> and 

3 this <a href=" http://web2py. com">a link</a></p> 

The markmin syntax is described in this file that ships with web2py: 
1 http : //127 . 0.0 . 1 : 8000/examples/static/markmin . html 

and some examples are shown in chapter 12 in the context of plugin_wiki, 
which uses MARKMIN extensively. 

You can use markmin to generate HTML, LaTeX and PDF documents: 

1 m = "Hello **world** [[link http://web2py.com] ]" 

2 from gluon.contrib. markmin. markmin2html import markmin2html 

3 print markmin2html(m) 

4 from gluon. contrib. markmin. markmin2latex import markmin2latex 

5 print markmin2latex(m) 

6 from gluon. contrib. markmin. markmin2pdf import markmin2pdf 

7 print markmin2pdf (m) # requires pdflatex 


(the MARKMIN helper is a shortcut for markmin2html) 
Here is a basic syntax primer: 
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SOURCE OUTPUT 


# title title 

## section section 

### subsection subsection 

**bold** bold 

"italic" italic 

"verbatim" verbatim 

http : //google . com http://google.com 

http ://... <a href="http://. . .">http:. . .</a> 

http://. . . png <img src="http:// . . . png" /> 

http://. . .mp3 <audio src="http:// . . .mp3"x/audio> 

http://. . .mp4 <video src="http:// . . .mp4"x/video> 

qr:http:// . . . <a h ref=" http ://... "ximg src="qr code"/x/a> 

embed : http://. . . <iframe src="http://. . . "x/if rame> 

[[click me #myanchor]] click me 


$$\int_ab sin(x)dx$$ J sin(x)dx 


Simply including a link to an image, a videos or an audio files without 
markup result in the corresponding image, video or audio file beling 
included automatically (for audio and video it uses html <audio> and 
<video> tags). 

Adding a link with the q r : prefix such as 
qr: http://web2py.com 

results in the corresponding QR code being embedded and linking the said 
URL. 

Adding a link tith the embed : prefix such as 
embed : http : //www. youtube . com/embed/xlw8hKTJ2Co 

results in the page being embedded, in this case a youtube video is 
embedded. 

Images can also be embedded with the following syntax: 
[[image-description http://. . ./image. png right 200px]] 


Unordered lists with: 

- one 

- two 

- three 


Ordered lists with: 


222 WEB2PY FULL-STACK WEB FRAMEWORK, 4TH EDITION 


1 + one 

2 + two 

3 + three 

and tables with: 

2 x 1 1 

3 1 x 1 

4 I I 1 

5 

The MARKMIN syntax also supports blockquotes, HTML5 audio and video 
tags, image alignment, custom ess, and it can be extended: 

1 MARKMIN( "' ' abab' ' -.custom" , extra=dict(custom=lambda text: text . replace) 'a' , 'c' ) ) 

generates 

'cbcb' 

Custom blocks are delimited by ". . .":<key> and they are rendered by the 
function passed as value for the corresponding key in the extra dictionary 
argument of MARKMIN. Mind that the function may need to escape the 
output to prevent XSS. 

OBJECT 

Used to embed objects (for example, a flash player) in the HTML. 

, >» print OBJECT! ' <hel lo>' , XHL( ' <b>world</b>^ ) , 

2 >» src=' http://www.web2py.com' ) 

3 <object src="http://www.web2py. com" >&lt;hello&gt;<t»world</t»</object> 

OL 

It stands for Ordered List. The list should contain LI tags. OL arguments that 
are not LI objects are automatically enclosed in <li>. . .</li> tags. 

■ >» print 0L( '<hello>' , XHL( '<b>world</b>' ) , _class=' test ' , _id=0) 

2 <ol id="0" class="rest"xli>&lt ;hello&gt;</li><lixb>world</b></li></ol> 

ON 

This is here for backward compatibility and it is simply an alias for True. It is 
used exclusively for checkboxes and deprecated since True is more Pythonic. 
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>» print INPUT (_type=' checkbox' , _name=' test ' , _checked=ON) 
<input checked^" checked" type=" checkbox" name="test" /> 

OPTGROUP 

Allows you to group multiple options in a SELECT and it is useful to 
customize the fields using CSS. 


>» print SELECT('a', 0PTGR0UP( '£>' , 

'c'» 

<select> 


<option value="a">a</option> 


<optgroup> 


<option value="fo">b</option> 


<option value="c'>c</option> 


</optgroup> 


</select> 



OPTION 

This should only be used as part of a SELECT/OPTION combination. 

>» print OPTION ( '<hello>' , XHL( '<b>world</b>' ) , _value='a') 
<option value="a">&lt;hello&gt ;<b>world</b></option> 

As in the case of INPUT, web2py make a distinction between "_value" (the 
value of the OPTION), and "value" (the current value of the enclosing select). 
If they are equal, the option is "selected". 

>» print SELECT('a', 'fa 1 , value='b'): 

<select> 

<option value="a">a</option> 

<option value="b" selected="selectecf">b</option> 

</select> 


This is for tagging a paragraph. 

1 >» print P( '<hello>' , XHL( '<b>world</b>' ) , _class=' test' , _id=0) 
z <p id="0" class="test">&lt ;hello&gt;<b>world</bx/p> 

PRE 

Generates a <pre>. . .</pre> tag for displaying pre-formatted text. The CODE 

helper is generally preferable for code listings. 

1 >» print PRE( '<hello>' , XML( '<b>world</b>' ) , _class=' test ' , _id=0) 
z <pre id="0" class="test">&lt ; hello&gt;<b>world</bx/pre> 


224 WEB2PY FULL-STACK WEB FRAMEWORK, 4TH EDITION 


SCRIPT 

This is include or link a script, such as JavaScript. The content between the 
tags is rendered as an HTML comment, for the benefit of really old browsers. 


1 

>» print SCRIPT!,' alert ("hello world");' 

, _language= 

'javascript' ) 

2 

<script language^" javascript"><\ - - 



3 

alert( "hello world"); 



4 

II- -></script> 




SELECT 

Makes a <select>. . .</select> tag. This is used with the OPTION helper. Those 
SELECT arguments that are not OPTION objects are automatically converted to 
options. 

, >» print SELECT('</?ello>' , XHL( ' <b>world<lb>' ) , _class=' test ' , _id=0) 

2 <select id="0" class="t"est"> 

3 <option value="&lt; hel lo&gt; ">&lt;hello&gt ;</option> 

4 <option value="&lt ; b&gt ;world&lt ; / b&gt ; "><b>world</bx/option> 

5 </select> 

SPAN 

Similar to DIV but used to tag inline (rather than block) content. 

, >» print SPAN('</iello>', XHL( '<b>world<lb>' ) , _class=' test ' , _id=0) 
z <span id="0" class="test">&lt ;hello&gt;<t»world</t»</span> 

STYLE 

Similar to script, but used to either include or link CSS code. Here the CSS is 
included: 



and here it is linked: 

>» print STYLE(_src='style. ess' ) 
<style src=" style. css"><\ -- 
//--></style> 


TABLE, TR, TD 
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These tags (along with the optional THEAD, TBODY and TFOOTER helpers) are used 
to build HTML tables. 

, >» print TABLE(TR(TD( a'), TD('b')), TR(TD('c'), TD('cT))) 

2 <tablextrxtd>a</tdxtd>b</tdx/trxtrxtd>c</tdxtd>d</tdx/trx/table> 

TR expects TD content; arguments that are not TD objects are converted 
automatically. 

, >» print TABLE(TR('a', '/)'), TR('c', 'd')) 

2 <table><t rxtd>a</tdxtd>b</tdx/t rxt rxtd>c</tdxtd>d</tdx/t rx/table> 

It is easy to convert a Python array into an HTML table using Python's * 
function arguments notation, which maps list elements to positional function 
arguments. 

Here, we will do it line by line: 

, >» table = [['a', '/?'], ['c' , 'd']] 

2 >» print TABLE(TR(*table[0]), TR(*table[l] )) 

3 <tablext rxtd>a</tdxtd>b</tdx/t rxt rxtd>c</tdxtd>d</tdx/t rx/table> 

Here we do all lines at once: 

, >» table = [['a' , l b l ], ['<_" , 'cf']] 

2 >» print TABLE (*[TR(* rows) for rows in table]) 

3 <tablextrxtd>a</tdxtd>b</tdx/trxtrxtd>c</tdxtd>d</tdx/trx/table> 

TBODY 

This is used to tag rows contained in the table body, as opposed to header or 
footer rows. It is optional. 

, >» print TBODY(TR('<ftello>'), _class=' test' , _id=0) 

2 <tbody id="0" class="test"xtrxtd>&lt;hello&gt ;</tdx/trx/tbody> 

TEXTAREA 

This helper makes a <textarea>. . .</textarea> tag. 

, >» print TEXTAREA (' <he 11 o>' , XHL( ' <b>world</b>' ) , _class=' test ' ) 

2 <textarea class="test" cols="40" rows="10">&lt;hello&gt;<b>world</bx/textarea> 

The only caveat is that its optional "value" overrides its content (inner HTML) 

1 >» print TEXTAREA(value="</ielZo world>" , _class="test") 

2 <textarea class="rest" cols="40" rows="I0">&lt; hello world&gt ;</textarea> 
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TFOOT 

This is used to tag table footer rows. 

, >» print TFOOT(TR(TD( •<hello> , )) l _class=' tesf , _id=0) 

2 <tfoot id="0" class="test"xtrxtd>&U;hello&gt;</tdx/trx/tfoot> 

TH 

This is used instead of TD in table headers. 

, >» print TH('</7elIo>' , XML ( , <b>world</b>' ) , _class=' tesf , _id=0) 
2 <th id="0" class="test">&lt ;hello&gt;<b>world</bx/th> 

THEAD 

This is used to tag table header rows. 

, >» print THEAD(TR(TH('<heno>')), _class=' test' , _id=0) 

2 <thead id="0" class="test"xtrxth>&lt;hello&gt ;</thx/trx/thead> 

TITLE 

This is used to tag the title of a page in an HTML header. 

, >» print TITLE (' <hel lo>' , XHL( ' <b>world</b>' ) ) 
2 <title>&lt ; hello&gt ; <t»wo rld</bx/title> 

TR 

Tags a table row. It should be rendered inside a table and contain 

<td>. . .</td> tags. TR arguments that are not TD objects will be automatically 

converted. 

, >» print TR( , <hello>' , XHL( , <b>world</b>' ) , _class=' test ' , _id=0) 

2 <tr id="0" class="rest"xtd>&lt ;hello&gt;</tdxtdxb>world</bx/tdx/tr> 


Tags text as typewriter (monospaced) text. 

>» print TT( '<hello>' , XML ( , <b>world</b>' ) , _class=' test ' , _id=0) 
<tt id="0" class="rest">&lt ;hello&gt;<b>world</bx/tt> 

UL 

Signifies an Unordered List and should contain LI items. If its content is not 
tagged as LI, UL does it automatically. 
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1 >» print UL( , <hello>' , XHL( , <b>world</b>' ) , _class=' test ' , _id=0) 

2 <ul id="0" class="t-est"xli>&lt ;hello&gt;</lixlixb>world</b></li></ul> 

embed64 

embed64(f ilename=None, file=None, data=None, extension='image/gif ' ) 
encodes the provided (binary) data into base64- filename: if provided, 
opens and reads this file in 'rb' mode, file: if provided, reads this file, data: 
if provided, uses the provided data. 

xmlescape 

xmlescape(data, quote=True) returns an escaped string of the provided data. 

1 >» print xmlescape ( '<hello>' ) 

2 &lt;heUo&gt; 


5.2.3 Custom helpers 


TAG 


Sometimes you need to generate custom XML tags. web2py provides TAG, a 
universal tag generator. 


{{=TAG.name 

!('a' , l b l , _c=' 

<*')}} 

generates 

the following 

XML 

<name c="d" 

>ab</name> 



Arguments "a", "b", and "d" are automatically escaped; use the XML helper to 
suppress this behavior. Using TAG you can generate HTML /XML tags not 
already provided by the API. TAGs can be nested, and are serialized with 
str( ) . An equivalent syntax is: 
{^TAGI'name'K'a 1 , 'b' , c='d')}} 

If the TAG object is created with an empty name, it can be used to concatenate 
multiple strings and HTML helpers together without inserting them into a 
surrounding tag, but this use is deprecated. Use the CAT helper instead. 

Notice that TAG is an object, and TAG. name or TAG [' name ' ] is a function that 
returns a temporary helper class. 


228 WEB2PY PULL-STACK WEB FRAMEWORK, 4TH EDITION 


MENU 

The MENU helper takes a list of lists or of tuples of the form of response . menu 
(as described in Chapter 4) and generates a tree-like structure using 
unordered lists representing the menu. For example: 

, >» print MENU ([['One' , False, •linkl'], ['Two 1 , False, 'link2 1 ]]) 

2 <ul class="weto2py-menu web2py-menu-vertical"> 

3 <lixa href="lin/<.Z">One</a></li> 

4 <lixa href="Iin/<2">Two</a></li> 

5 </ul> 

Each menu item can have a fourth argument that is a nested submenu (and 
so on recursively): 

, >» print HENU([[' One 1 , False, 'linkl', [{'Two' , False, 'link2']]]]) 

2 <ul class="weto2py-menu web2py-menu-vertical"> 

3 <li c\ass="web2py-menu-expand"> 

4 <a href="lin/<.Z">One</a> 

5 <ul class='Veb2py-menu-i/erticaZ"> 

6 <lixa href="lin(<2">Two</a></li> 

7 </ul> 
s </li> 

9 </ul> 

The MENU helper takes the following optional arguments: 

• _class: defaults to "web2py-menu web2py-menu-vertical" and sets the 
class of the outer UL elements. 

• ul_class: defaults to "web2py-menu-vertical" and sets the class of the inner 
UL elements. 

• li_class: defaults to "web2py-menu-expand" and sets the class of the inner 
LI elements. 

MENU takes an optional argument mobile. When set to True instead of building 
a recursive UL menu structure it returns a SELECT dropdown with all the menu 
options and a onchange attribute that redirects to the page corresponding to 
the selected option. This is designed an an alterantive menu representation 
that increases usability on small mobile devices such as phones. 

Normally the menu is used in a layout with the following syntax: 
1 {{=MENU( response. menu, mobile= request .user_agent( ) .is_mobile)}} 
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In this way a mobile device is automatically detected and the menu is 
rendered accordingly. 


5.3 BEAUTIFY 

BEAUTIFY is used to build HTML representations of compound objects, 
including lists, tuples and dictionaries: 

{{=BEAUTIFY({"a": ["hello", XML('VorlcT) ] , "fa": (1, 2)})}} 

BEAUTIFY returns an XML-like object serializable to XML, with a nice 
looking representation of its constructor argument. In this case, the XML 
representation of: 

{"a": ["hello", XHL( "world" )] , "fa": (1, 2)} 

will render as: 

<table> 

<trxtd>a</td><td>:</tdxtd>hello<br />world</tdx/tr> 

<t rxtd>b</tdxtd> : </tdxtd>l<b r />2</tdx/t r> 

</table> 


5.4 Server-side DOM and parsing 


5.4.I elements 

The DIV helper and all derived helpers provide the search methods element 
and elements. 

element returns the first child element matching a specified condition (or 
None if no match). 

elements returns a list of all matching children. 

element and elements use the same syntax to specify the matching condition, 
which allows for three possibilities that can be mixed and matched: jQuery- 
like expressions, match by exact attribute value, match using regular 
expressions. 
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Here is a simple example: 

>» a = DIV(DIV(DIV('a' , _id=' target' ,_class='abc' ) ) ) 

>» d = a. elements ( ' div#target' ) 

>» d[0] [0] = 'changed' 

>» print a 

<divxdivxdiv id=" target" class="abc">changed</div></divx/div> 

The un-named argument of elements is a string, which may contain: the name 
of a tag, the id of a tag preceded by a pound symbol, the class preceded by a 
dot, the explicit value of an attribute in square brackets. 

Here are 4 equivalent ways to search the previous tag by id: 



Here are 4 equivalent ways to search the previous tag by class: 



Any attribute can be used to locate an element (not just id and class), 
including multiple attributes (the function element can take multiple named 
arguments), but only the first matching element will be returned. 

Using the jQuery syntax "div#target" it is possible to specify multiple search 
criteria separated by a space: 


»> a = DIV(SPAN('a' , _id=' tl ' ) ,DIV( 'b' ,_class='c2' ) ) 
>» d = a. elements ( ' span#t 1, div#c2') 

or equivalently 

>» a = DIV(SPAN('a' , _id=' tl ' ) ,DIV( 'b' ,_class='c2' ) ) 
>» d = a. elements! ' span#tl ' , 'div#c2' ) 

If the value of an attribute is specified using a name argument, it can be a 
string or a regular expression: 

>» a = DIV(SPAN('a' , _id= ' testl23 ' ) , DIV( 'b' ,_class='c2' ) ) 

>» d = a. elements ( 'span' , _id=re. compile! ' test\d{3} ' ) 
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A special named argument of the DIV (and derived) helpers is find. It can 
be used to specify a search value or a search regular expression in the text 
content of the tag. For example: 

»> a = DIV ( SPAN ['abcde'), DIV (' fghij ' )) 
>» d = a. elements (find=' bed' ) 
>» print d[0] 
<span>abcde</span> 


or 



5.4.2 components 


Here's an example of listing all elements in an html string: 

html = TAG( '<a>xxx</axb>yyy</b>' ) 

for item in html. components: print item 


5.4.3 parent 


parent returns the parent of the current element. 

, >» a = DIV(SPAN('a'),DIV('b')) 

2 >» d = a. element! 'a ') .parent ( ) 

3 >» d[ ' . class' ] = ' abc' 

4 >» print a 

5 <div class="atoc"><span>a</span><div>b</div></div> 


^.4.4 flatten 

The flatten method recursively serializes the content of the children of a given 
element into regular text (without tags): 


, >» a = DIV ( SPAN (' this' , DlVCis', B('a'))), SPANftest' 

2 >» print a. flatten!) 

3 thisisatest 
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Flatten can be passed an optional argument, render, i.e. a function that 
renders /flattens the content using a different protocol. Here is an example 
to serialize some tags into Markmin wiki syntax: 

, >» a = DIV(H1( 'title'), PC example of a ' , A('Zinfc', _href='#test ' ) ) ) 

2 >» from gluon.html import markmin_serializer 

3 >» print a. flatten! render=markmin_serializer) 

4 # titles 

5 

6 example of [[a link #test]] 

At the time of writing we provide markmin_serializer and 
ma rkdown_ serializes 


5.4.5 Parsing 

The TAG object is also an XML/HTML parser. It can read text and convert 

into a tree structure of helpers. This allows manipulation using the API 

above: 

>» html = '<hl>Title</hlxp>this is a <span>test</spanx/p>' 

>» parsed_html = TAG(html) 

>» parsed. html .element) 'span' ) [0] = ' TEST' 

>» print parsed_html 

<hl>Title</hlxp>this is a <span>TEST</spanx/p> 


5.5 Page layout 

Views can extend and include other views in a tree-like structure. 

For example, we can think of a view "index.html" that extends "layout.html" 
and includes "bodyhtml". At the same time, "layout.html" may include 
"header.html" and "footer.html". 

The root of the tree is what we call a layout view. Just like any other HTML 
template file, you can edit it using the web2py administrative interface. The 
file name "layout.html" is just a convention. 

Here is a minimalist page that extends the "layout.html" view and includes 
the "page.html" view: 
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{{extend 

layou; 

i.html 

'}} 

<hl>HeUo 

World</hl> 


{{include 

■page. 

html' 

}} 


The extended layout file must contain an {{include}} directive, something 
like: 

1 <html> 

2 <head> 

3 <title>Page Title</title> 

4 </head> 

5 <body> 

6 {{include}} 
- </body> 

s </html> 

When the view is called, the extended (layout) view is loaded, and the 
calling view replaces the {{include}} directive inside the layout. Processing 
continues recursively until all extend and include directives have been 
processed. The resulting template is then translated into Python code. Note, 
when an application is bytecode compiled, it is this Python code that is 
compiled, not the original view files themselves. So, the bytecode compiled 
version of a given view is a single. pyc file that includes the Python code not 
just for the original view file, but for its entire tree of extended and included 
views. 

extend, include, block and super are special template directives, not Python 
commands. 

Any content or code that precedes the {{extend ...}} directive will be 
inserted (and therefore executed) before the beginning of the extended view's 
content /code. Although this is not typically used to insert actual HTML 
content before the extended view's content, it can be useful as a means to 
define variables or functions that you want to make available to the extended 
view. For example, consider a view "index.html": 

1 {{sidebar_enabled=True}} 

2 {{extend 'layout.html'}} 

3 <hl>Home Page</hl> 

and an excerpt from "layout.html": 
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1 {{if sidebar_enabled: }} 

2 <div id="sidebar"> 

3 Sidebar Content 

4 </div> 

5 {{pass}} 


Because the sidebar_enabled assignment in "index. html" comes before the 
extend, that line gets inserted before the beginning of "layout.html", 
making sidebar_enabled available anywhere within the "layout.html" code (a 
somewhat more sophisticated version of this is used in the welcome app). 

It is also worth pointing out that the variables returned by the controller 
function are available not only in the function's main view, but in all of its 
extended and included views as well. 

The argument of an extend or include (i.e., the extended or included view 
name) can be a python variable (though not a python expression). However, 
this imposes a limitation - views that use variables in extend or include 
statements cannot be bytecode compiled. As noted above, bytecode compiled 
views include the entire tree of extended and included views, so the specific 
extended and included views must be known at compile time, which is not 
possible if the view names are variables (whose values are not determined 
until run time). Because bytecode compiling views can provide a significant 
speed boost, using variables in extend and include should generally be 
avoided if possible. 

In some cases, an alternative to using a variable in an include is simply to 
place regular {{include . . .}} directives inside an if . . .else block. 



The above code does not present any problem for bytecode compilation 
because no variables are involved. Note, however, that the bytecode compiled 
view will actually include the Python code for both "this_view.html" and 
"that_view.html", though only the code for one of those views will be 
executed, depending on the value of some_condition. 


THE VIEWS 235 


Keep in mind, this only works for include - you cannot place {{extend . . . }} 
directives inside if . . .else blocks. 

Layouts are used to encapsulate page commonality (headers, footers, menus), 
and though they are not mandatory, they will make your application easier 
to write and maintain. In particular, we suggest writing layouts that take 
advantage of the following variables that can be set in the controller. Using 
these well known variables will help make your layouts interchangeable: 

response. title 
response. subtitle 
response . met a .author 
response. met a. keywords 
response . meta . desc ription 
response. flash 
response. menu 
response. files 

Except for menu and files, these are all strings and their meaning should be 
obvious. 

response. menu menu is a list of 3-tuples or 4-tuples. The three elements are: 
the link name, a boolean representing whether the link is active (is the current 
link), and the URL of the linked page. For example: 

response. menu = [('Google' , False, ' http://www.google.com' ,[]) , 
{'Index', True, URL( 'index' ) , [])] 


The fourth tuple element is an optional sub-menu. 

response, files is a list of CSS and JS files that are needed by your page. 

We also recommend that you use: 

{{include 'web2py-ajax.html'}} 

in the HTML head, since this will include the jQuery libraries and define 
some backward-compatible JavaScript functions for special effects and Ajax. 
"web2py_ajax.html" includes the response. meta tags in the view, jQuery base, 
the calendar datepicker, and all required CSS and JS response. files. 
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5.5.1 Default page layout 

Below is the "views/layout.html" that ships with the web2py scaffolding 
application welcome (stripped down of some optional parts). Any new 
application will have a similar default layout: 

■ <!D0CTYPE html> 

2 <head> 

3 <meta charset="utf-8" /> 

4 <title>{{=response. title or request .application}}</title> 

5 

6 <!-- http://dev.w3.org/html5/markup/meta.name.html --> 

7 <meta name=" application- name" content^" {{^request .application}}" /> 
a 

,., <script s rc="{" {=URL( l static ' , ' js/modernizr. custom. js' )}}"></script> 
10 

u <!-- include stylesheets --> 

,4 {{ 

13 response. files. append ( URL ( 'static' , ' ess/skeleton. ess ' ) ) 

14 response. files. append ( URL ( 'static' , ' css/web2py. ess' ) ) 

15 response. files. append ( URL ( 'static' , ' css/superfish. ess ' ) ) 
iB response. files. append ( URL ( 'static' , ' js/superfish. js ' ) ) 

T }} 

18 

n) {{include 'web2py-ajax.html'}} 

20 

41 <script type="text/jai/ascript"> 

22 j Query (function! ){ jQuery( ' ul. sf-menu' ) . supersubs({minWidth: 12,maxWidth:30, 

extraWidth:3}).superfish(); }); 

23 </script> 
24 

=5 {{ 

40 # using sidebars need to know what sidebar you want to use 

27 left_sidebar_enabled = globals( ) .get( ' leftsidebar-enabled' , False) 

2S right_sidebar_enabled = globals( ) .get( ' right_sidebar_enabled' , False) 

41) middle_columns = {0: 'sixteen' , 1: ' twelve' ,2: 'eight '} [ 

3 o (left_sidebar_enabled and 1 or 0)+( right_sidebar_enabled and 1 or 0)] 

3. }} 

32 

33 </head> 

34 <body> 

35 <div class='Vrapper"><! - - for sticky footer --> 

3<i 

37 <div class="topbar"> 

3 s <div class="container"> 

39 <div class="sixteen columns"> 

40 <div id="navbar"> 

41 {{='auth l in globals() and auth. navbar(separators=( ' ',' | ',''))}} 
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42 </div> 

43 <div id="menu"> 

44 {{=MENU( response. menu, 

45 _class='motoiIe-menu' if is_mobile else 'sf-menu' 
4 (, mobile=request .user_agent ( ) . is_mobile)}} 

47 </div> 

48 </div> 

49 </div> 

50 </div><!-- topbar --> 

51 

52 <div class="flash">{{=response. flash or ''}}</div> 

53 

54 <div class="/7eacfer"> 

55 <div class="container"> 

56 <div class="sixteen columns"> 

57 <hl class=" remove-bottom" sty\e=" margin- top: .5em;"> 

58 {{^response. title or request .application}} 

59 </hl> 

60 <h5>{{=response. subtitle or ''}}</h5> 

61 </div> 
62 

63 <div class="sixteen columns"> 

64 <div class="statusbar"> 

65 {{block statusbar}} 

66 <span class="fcreadcri/mbs">{{=request .function}}</span> 

67 {{end}} 

68 </div> 

69 </div> 

70 </div> 

71 </div> 
72 

73 <div class="main"> 

74 <div class="container"> 

75 {{if left_sidebar_enabled: }} 

76 <div class="four columns left-sidebar"> 

77 {{block left_sidebar}} 

78 <h3>Left Sidebar</h3> 

79 <px/p> 
so {{end}} 

81 </div> 

82 {{pass}} 
83 

84 <div class="{{=middle_columns» columns center"> 

85 {{block center}} 

86 {{include}} 

87 {{end}} 
ss </div> 

89 

90 {{if right_sidebar_enabled: }} 


91 
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<div class=" four columns"> 

92 {{block right_sidebar}} 

93 <h3>Right Sidebar</h3> 

94 <px/p> 

95 {{end}} 

96 </div> 

97 {{pass}} 
98 

99 </div><!-- container --> 

00 </div><!-- main --> 

02 <div class="pi/sh"x/div> 

03 </div><!-- wrapper --> 
04 

05 <div class=" footer"> 

06 <div class="conrainer header"> 

07 <div class="sixteen columns"> 

08 {{block footer}} <!-- this is default footer --> 

09 <div class="footer-contenr" > 

{{=!{.' Copyright')}} &#169; 2011 

u <div style=" float: right;"> 

12 <a href="http://www. web2py.com/"> 

13 <img style="padding-bottom: 0; " 

i 4 src="{{=URL( 'static' , ' images/poweredby.png' )}}"/> 

15 </a> 

16 </div> 

17 </div> 
is {{end}} 

19 </div> 

20 </div><!-- container --> 

21 </div><!-- footer --> 
22 

2, </body> 

24 </html> 


There are a few features of this default layout that make it very easy to use 
and customize: 

• It is written in HTML5 and uses the "modernizr" [49] library for backward 
compatibility. The actual layout include some extra conditional statements 
required by IE and they are omitted for brevity. 

• It displays both response. title and response, subtitle which can be set in 
a model. If they are not set, it adopts the application name as title 

• It includes the web2py_ajax. html file in the header which generated all the 
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link and script import statements. 

• It uses a modified version of "skeleton" [50] a library for flexible layouts 
which works on mobile devices and re-arranges columns to fit small 
screened. 

• It uses "superfish.js" for dynamic cascading menus. There is an explicit 
script to activate the superfish cascading menu and it can be removed if 
not necessary. 

• The {{=auth.navbar( . . . )}} displays a welcome to the current user 
and links to auth functions like login, logout, register, change 
password, etc. depending on context. It is a heler factory and its 
output can be manipulated as any other helper. It is placed in a 
{{try : }} . . . {{except : pass}} in case auth is undefined. 

• The {{=MENU( response. menu) displays the menu structure as <ul>. . .</ul>. 

• {{include}} is replaced by the content of the extending view when the 
page is rendered. 

• By default it uses a conditional three column (the left and right sidebars 
can be turned off by the extending views) 

• It uses the following classes: header, main, footer 

• It contains the following blocks: statusbar, left_sidebar, center, 
right_sidebar, footer. 

Views can turn on and fill sidebars as follows: 


1 {{left_sidebar_enable=True}} 

2 {{extend 'layout.html'}} 
3 

4 This text goes in center 
5 

6 {{block left.sidebar}} 

7 This text goes in sidebar 
x {{end}} 
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5.5.2 Customizing the default layout 

Customizing the default layout without editing is very easy because the CSS 
files are documented: 

• "skeleton. ess" contains reset, grid layout and form styles 

• "web2pycss" contains web2py specific styles 

• "superfish.css" contains menu styles 

To change colors and background images, simply append the following code 
to "web2pycss": 

body { background: url(' images/ background. prig' ) repeat -x #3 A3 A3 A; } 

a { color: #349001; } 

.header hi { color: #349001; } 

.header h2 { color: white; font-style: italic; font-size: 14px;} 

.statusbar { background: #333333; border- bottom: 5px #349001 solid; } 

.statusbar a { color: white; } 

.footer { border-top: 5px #349001 solid; } 

The menu is built in a color-neutral way but you can change that too. 

Of course you can also completely replace the "layout.html" and "web2pycss" 
files with your own. 

5.5.3 Mobile development 

The default layout.html is designed to be friendly to mobile devices but that 
is not enought. One may need to use different views when a page is visited 
by a mobile device. 

To make developing for desktop and mobile devices easier, web2py includes 
the @mobilize decorator. This decorator is applied to actions that should have 
a normal view and a mobile view. This is demonstrated here: 


1 from gluon. contrib.user_agent_parser import mobilize 

2 (amobilize 

3 def index ( ) : 

4 return dict() 


Notice that the decorator must be important once before using it in a 
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controller. When the "index" function is called from a regular browser 
(desktop computer), web2py will render the returned dictionary using the 
view "[controller] /index.html". However, when it is called by a mobile 
device, the dictionary will be rendered by "[controller] /index. mobile.html". 
Notice that mobile views have the "mobile.html" extension. 

Alternatively you can apply the following logic to make all views mobile 
friendly: 

if request. user„agent( ) .is_mobile: 

response. view. replace ( ' .html' , ' .mobile.html ' ) 

The task of creating the "*. mobile. html" views is left to the developer but we 
strongly suggest using the "jQuery Mobile" plugin which makes the task very 
easy. 


5.6 Functions in views 


Consider this "layout.html": 

<html> 
<body> 

{{include}} 

<div class="side£>ar"> 

{{if 'mysidebar' in globals( ) :}}{{mysidebar( )}}{{else: }} 

my default sidebar 
{{pass}} 
</div> 
</body> 
</html> 


and this extending 

view 

{{def mysidebar() 

:}} 


my new sidebar! ! ! 



{{return}} 



{{extend 'layout. 

html 

'}} 

Hello World! ! ! 




Notice the function is defined before the {{extend . . . }} statement - this results 
in the function being created before the "layout.html" code is executed, so 
the function can be called anywhere within "layout.html", even before the 
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{{include}}. Also notice the function is included in the extended view 
without the = prefix. 

The code generates the following output: 

1 <html> 

2 <body> 

3 Hello World! ! ! 

4 <div class=" sidebar"> 

5 my new sidebar! ! ! 

6 </div> 
- </body> 

8 </html> 

Notice that the function is defined in HTML (although it could also contain 
Python code) so that response. write is used to write its content (the function 
does not return the content). This is why the layout calls the view function 
using {{mysidebar( )}} rather than {{=mysidebar( )}}. Functions defined in this 
way can take arguments. 


5. j Blocks in views 

Another way to make a view more modular is by using {{block... }}s and 
this mechanism is an alternative to the mechanism discussed in the previous 
section. 

Consider this "layout.html": 

<html> 
<body> 

{{include}} 
<div class="sidebar"> 
{{block mysidebar}} 

my default sidebar 
{{end}} 
</div> 
</body> 
</html> 


and this extending view 

{{extend 'layout.html'}} 
Hello World! ! ! 
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, {{block mysidebar}} 

4 my new sidebar! ! ! 

5 {{end}} 

It generates the following output: 

1 <html> 

2 <body> 

3 Hello World! ! ! 

4 <div class=" sidebar"> 

5 my new sidebar! ! ! 

6 </div> 
- </body> 

s </html> 

You can have many blocks, and if a block is present in the extended view but 
not in the extending view, the content of the extended view is used. Also, 
notice that unlike with functions, it is not necessary to define blocks before 
the {{extend ...}}- even if defined after the extend, they can be used to make 
substitutions anywhere in the extended view. 

Inside a block, you can use the expression {{super}} to include the content of 
the parent. For example, if we replace the above extending view with: 

1 {{extend ' layout. html ' }} 

2 Hello World! ! ! 

, {{block mysidebar}} 

4 {{super}} 

5 my new sidebar! ! ! 

6 {{end}} 

we get: 

1 <html> 

2 <body> 

3 Hello World! ! ! 

4 <div class="sidebar"> 

5 my default sidebar 

6 my new sidebar! ! ! 

7 </div> 

8 </body> 

9 </html> 


The database abstraction layer 


6.1 Dependencies 


web2py comes with a Database Abstraction Layer (DAL), an API that maps 
Python objects into database objects such as queries, tables, and records. 
The DAL dynamically generates the SQL in real time using the specified 
dialect for the database back end, so that you do not have to write SQL 
code or learn different SQL dialects (the term SQL is used generically), and 
the application will be portable among different types of databases. At the 
time of this writing, the supported databases are SQLite (which comes with 
Python and thus web2py), PostgreSQL, MySQL, Oracle, MSSQL, FireBird, 
DB2, Informix, and Ingres and (partially) the Google App Engine (SQL and 
NoSQL). Experimentally we support more databases. Please check on the 
web2py web site and mailing list for more recent adapters. Google NoSQL is 
treated as a particular case in Chapter 13. 

The Windows binary distribution works out of the box with SQLite and 
MySQL. The Mac binary distribution works out of the box with SQLite. To 
use any other database back-end, run from the source distribution and install 
the appropriate driver for the required back end. 

Once the proper driver is installed, start web2py from source, and it will find 
the driver. Here is a list of drivers: 
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database driver (source) 

SQLite sqlite3 or pysqlite2 or zxJDBC [58] (on Jython) 

PostgreSQL psycopg2 [59] or zxJDBC [58] (on Jython) 

MySQL pymysql [60] or MySQLdb [61] 

Oracle cx_Oracle [62] 

MSSQL pyodbc [63] 

FireBird kinterbasdb [64] 

DB2 pyodbc [63] 

Informix informixdb [65] 

Ingres ingresdbi [66] 

(pymysql ships with web2py) web2py defines the following classes that make 
up the DAL: 

DAL represents a database connection. For example: 
db = DAL( ' sqlite:// storage. db' ) 

Table represents a database table. You do not directly instantiate Table; 
instead, DAL.define_table instantiates it. 

db.define_table( 'mytable' , Field ( 'my field' ) ) 

The most important methods of a Table are: .insert, .truncate, .drop, and 
. import_f rom_csv_f ile. 

Field represents a database field. It can be instantiated and passed as an 
argument to DAL.define_table. 

DAL Rows is the object returned by a database select. It can be thought of 
as a list of Row rows: 

rows = db(db. mytable. myfield!=None) .select( ) 

Row contains field values. 


1 for row in rows: 

2 print row.myfield 


Query is an object that represents a SQL "where" clause: 
myquery = (db. mytable. myfield != None) | (db. mytable. myfield > 'A') 

Set is an object that represents a set of records. Its most important methods 
are count, select, update, and delete. For example: 
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myset = db(myquery) 

rows = myset . select) ) 

myset . update (myfield= ' somevalue ' ) 

myset .delete ( ) 

Expression is something like an orderby or groupby expression. The Field 

class is derived from the Expression. Here is an example. 

myorder = db.mytable.myfield. upper( ) | db.mytable.id 
db( ) .select (db. table. ALL, orderby=myorder) 


6.2 Connection strings 

A connection with the database is established by creating an instance of the 

DAL object: 

>» db = DAL( ' sqlite: //storage. db' , pool_size=0) 

db is not a keyword; it is a local variable that stores the connection object DAL. 
You are free to give it a different name. The constructor of DAL requires 
a single argument, the connection string. The connection string is the 
only web2py code that depends on a specific back-end database. Here 
are examples of connection strings for specific types of supported back-end 
databases (in all cases, we assume the database is running from localhost on 
its default port and is named "test"): 

SQLite sqlite://storage.db 

MySQL mysql ://username: pas sword@local host/test 

PostgreSQL postgres://username: pas sword@localhost/ test 

MSSQL mssql : //username: pas sword@local host/test 

FireBird firebird : //username : password@localhost/test 

Oracle oracle: //use rname/passwordgtest 

DB2 db2 ://username: password@test 

Ingres ing res : //username: pa ssword@local host/test 

Informix informix: //username : password@test 

Google App Engine/SQL google : sql 

Google App Engine/NoSQL google: datasto re 

Notice that in SQLite the database consists of a single file. If it does not 
exist, it is created. This file is locked every time it is accessed. In the case 
of MySQL, PostgreSQL, MSSQL, FireBird, Oracle, DB2, Ingres and Informix 
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the database "test" must be created outside web2py. Once the connection is 
established, web2py will create, alter, and drop tables appropriately. 

It is also possible to set the connection string to None. In this case DAL will 
not connect to any back-end database, but the API can still be accessed for 
testing. Examples of this will be discussed in Chapter 7. 


6.2.1 Connection pooling 

The second argument of the DAL constructor is the pool_size; it defaults to 

o. 

As it is rather slow to establish a new database connection for each request, 
web2py implements a mechanism for connection pooling. Once a connection 
is established and the page has been served and the transaction completed, 
the connection is not closed but goes into a pool. When the next http request 
arrives, web2py tries to obtain a connection from the pool and use that for 
the new transaction. If there are no available connections in the pool, a new 
connection is established. 

The pool_size parameter is ignored by SQLite and Google App Engine. 

Connections in the pools are shared sequentially among threads, in the sense 
that they may be used by two different but not simultaneous threads. There 
is only one pool for each web2py process. 

When web2py starts, the pool is always empty. The pool grows up to the 
minimum between the value of pool_size and the max number of concurrent 
requests. This means that if pool_size=10 but our server never receives more 
than 5 concurrent requests, then the actual pool size will only grow to 5. If 
pool_size=Q then connection pooling is not used. 

Connection pooling is ignored for SQLite, since it would not yield any 
benefit. 
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6.2.2 Connection failures 

If weD2py fails to connect to the database it waits 1 seconds and tries again 
up to 5 times before declaring a failure. In case of connection pooling it is 
possible that a pooled connection that stays open but unused for some time 
is closed by the database end. Thanks to the retry feature web2py tries to 
re-establish these dropped connections. 

When using connection pooling a connection is used, put back in the pool 
and then recycled. It is possible that while the connection is idle in pool 
the connection is closed by the database server. This can be because of a 
malfunction or a timeout. When this happens web2py detects it and re- 
establish the connection. 

6.2.3 Replicated databases 

The first argument of DAL( . . . ) can be a list of URIs. In this case web2py 
tries to connect to each of them. The main purpose for this is to deal with 
multiple database servers and distribute the workload among them). Here is 
a typical use case: 
db = DkLd 1 mysql: //... 1' , 'mysql:// . . .2' , 'mysql:// . . .3']) 

In this case the DAL tries to connect to the first and, on failure, it will try the 
second and the third. This can also be used to distribute load in a database 
master-slave configuration. We will talk more about this in Chapter 13 in the 
context of scalability. 

6.3 Reserved keywords 

There is also another argument that can be passed to the DAL constructor 
to check table names and column names against reserved SQL keywords in 
target back-end databases. 

This argument is check_ reserved and it defaults to None. 

This is a list of strings that contain the database back-end adapter names. 
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The adapter name is the same as used in the DAL connection string. So if you 

want to check against PostgreSQL and MSSQL then your connection string 

would look as follows: 

db = DAL( ' sqlite:// storage, db' , 

check- reserved=[ ' postgres ' , ' mssql ' ] ) 

The DAL will scan the keywords in the same order as of the list. 

There are two extra options "all" and "common". If you specify all, it will 
check against all known SQL keywords. If you specify common, it will only 
check against common SQL keywords such as SELECT, INSERT, UPDATE, etc. 

For supported back-ends you may also specify if you would like to check 
against the non-reserved SQL keywords as well. In this case you would 
append _non reserved to the name. For example: 

check_reserved=[ 'postgres' , ' postgres-nonreserved' ] 

The following database backends support reserved words checking. 


PostgreSQL postgres (_non reserved) 


MySQL 

mysql 

FireBird 

firebird (_non reserved) 

MSSQL 

mssql 

Oracle 

oracle 


6.4 DAL, Table, Field 

The best way to understand the DAL API is to try each function yourself. 
This can be done interactively via the web2py shell, although ultimately, DAL 
code goes in the models and controllers. 

Start by creating a connection. For the sake of example, you can use SQLite. 
Nothing in this discussion changes when you change the back-end engine. 
>» db = DAL( ' sqlite: //storage. db' ) 

The database is now connected and the connection is stored in the global 
variable db. 

At any time you can retrieve the connection string. 
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>» print db._uri 
sqlite://storage.db 

and the database name 

>» print db._dbname 
sqlite 

The connection string is called a _uri because it is an instance of a Uniform 
Resource Identifier. 

The DAL allows multiple connections with the same database or with 
different databases, even databases of different types. For now, we will 
assume the presence of a single database since this is the most common 
situation. 

The most important method of a DAL is def ine_ table: 
>» db.define_table( 'person' , Field ( 'name' ) ) 

It defines, stores and returns a Table object called "person" containing a field 
(column) "name". This object can also be accessed via db. person, so you do 
not need to catch the return value. 

Do not declare a field called "id", because one is created by web2py anyway. 
Every table has a field called "id" by default. It is an auto-increment integer 
field (starting at 1) used for cross-reference and for making every record 
unique, so "id" is a primary key. (Note: the id's starting at 1 is back-end 
specific. For example, this does not apply to the Google App Engine NoSQL.) 

Optionally you can define a field of type='id' and web2py will use this field 
as auto-increment id field. This is not recommended except when accessing 
legacy database tables. With some limitation, you can also use different 
primary keys and this is discussed in the section on "Legacy databases and 
keyed tables". 


6.5 Record representation 

It is optional but recommended to specify a format representation for records: 
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>» db.define_table( 'person' , Field (' name ') , format= 

'%(name)s' 

) 

or 

>» db.define_table( 'person' , Field (' name ') , format= 

'%(name)s 

*(id)s') 

or even more complex ones using a function: 

>» db.define_table( 'person' , Field ( 'name' ) , 

format=lambda r: r.name or 'anonymous') 


The format attribute will be used for two purposes: 

• To represent referenced records in select/option drop-downs. 

• To set the db . othertable . person . represent attribute for all fields referencing 
this table. This means that SQLTABLE will not show references by id but 
will use the format preferred representation instead. 

These are the default values of a Field constructor: 

Field(name, 'string', length=None, default=None, 
required=False, requires^' <default>' , 
ondelete=' CASCADE' , notnull=False, unique=False, 
uploadfield=True, widget=None, label=None, comment=None : 
writable=True, readable=True, update=None, authorize=None, 
autodelete=False, represent=None, compute=None, 
uploadfolder=os.path. join (request .folder, ' uploads' ) , 
uploadseparate=None) 

Not all of them are relevant for every field, "length" is relevant only for fields 
of type "string", "uploadfield" and "authorize" are relevant only for fields of 
type "upload", "ondelete" is relevant only for fields of type "reference" and 
"upload". 

• length sets the maximum length of a "string", "password" or "upload" field. 
If length is not specified a default value is used but the default value is not 
guaranteed to be backward compatible. To avoid unwanted migrations on 
upgrades, we recommend that you always specify the length for string, password 
and upload fields. 

• default sets the default value for the field. The default value is used when 
performing an insert if a value is not explicitly specified. It is also used 
to pre-populate forms built from the table using SQLFORM. Note, rather 
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than being a fixed value, the default can instead be a function (including a 
lambda function) that returns a value of the appropriate type for the field. 
In that case, the function is called once for each record inserted, even when 
multiple records are inserted in a single transaction. 

• required tells the DAL that no insert should be allowed on this table if a 
value for this field is not explicitly specified. 

• requires is a validator or a list of validators. This is not used by the DAL, 
but it is used by SQLFORM. The default validators for the given types are 
shown in the following table: 

field type default field validators 

string IS_LENGTH(length) default length is 512 

text IS_LENGTH(65536) 

blob None 

boolean None 

integer IS_INT_IN_RANGE( -lelOO, lelOO) 

double IS_FLOAT_IN_RANGE(-lel00, lelQO) 

decimal (n,m) IS_DECIMAL_IN_RANGE( -lelOO, lelOO) 

date IS_DATE() 

time IS_TIME() 

datetime IS_DATETIME( ) 

password None 

upload None 

reference <table> IS_II\LDB(db, table. field, format ) 

list:string None 

list:integer None 

list: reference <table> IS_IN_DB(db,table.field, format , mult iple=T rue) 

Decimal requires and returns values as Decimal objects, as defined in the 
Python decimal module. SQLite does not handle the decimal type so internally 
we treat it as a double. The (n,m) are the number of digits in total and the 
number of digits after the decimal point respectively. 

The list: fields are special because they are designed to take advantage 
of certain denormalization features on NoSQL (in the case of Google App 
Engine NoSQL, the field types ListProperty and StringListProperty) and 
back-port them all the other supported relational databases. On relational 
databases lists are stored as a text field. The items are separated by a | and 
each I in string item is escaped as a | | . They are discussed in their own 
section. 
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Notice that requires=. . . is enforced at the level of forms, required=True is 
enforced at the level of the DAL (insert), while notnull, unique and ondelete 
are enforced at the level of the database. While they sometimes may seem 
redundant, it is important to maintain the distinction when programming with 
the DAL. 

ondelete translates into the "ON DELETE" SQL statement. By default it is 
set to "CASCADE". This tells the database that when it deletes a record, 
it should also delete all records that refer to it. To disable this feature, set 
ondelete to "NO ACTION" or "SET NULL". 

notnull=True translates into the "NOT NULL" SQL statement. It prevents 
the database from inserting null values for the field. 

unique=True translates into the "UNIQUE" SQL statement and it makes sure 
that values of this field are unique within the table. It is enforced at the 
database level. 

uploadf ield applies only to fields of type "upload". A field of type "upload" 
stores the name of a file saved somewhere else, by default on the filesystem 
under the application "uploads/" folder. If uploadfield is set, then the file 
is stored in a blob field within the same table and the value of uploadfield 
is the name of the blob field. This will be discussed in more detail later in 
the context of SQLFORM. 

uploadfolder defaults to the application's "uploads/" folder. If set to a 
different path, files will uploaded to a different folder. For example, 
uploadfolder=os.path.join(request. folder/static/temp') will upload files to 
the web2py/ applications /myapp/ static /temp folder. 

uploadseparate if set to True will upload files under different subfolders 
of the uploadfolder folder. This is optimized to avoid too many files under 
the same folder /sub folder. ATTENTION: You cannot change the value of 
uploadseparate from True to False without breaking the system. web2py 
either uses the separate subfolders or it does not. Changing the behavior 
after files have been uploaded will prevent web2py from being able to 
retrieve those files. If this happens it is possible to move files and fix the 
problem but this is not described here. 
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widget must be one of the available widget objects, including custom 
widgets, for example: SQLFORM. widgets. string. widget. A list of available 
widgets will be discussed later. Each field type has a default widget. 

label is a string (or something that can be serialized to a string) that 
contains the label to be used for this field in autogenerated forms. 

comment is a string (or something that can be serialized to a string) that 
contains a comment associated with this field, and will be displayed to the 
right of the input field in the autogenerated forms. 

writable if a field is writable, it can be edited in autogenerated create and 
update forms. 

readable if a field is readable, it will be visible in readonly forms. If a 
field is neither readable nor writable, it will not be displayed in create and 
update forms. 

update contains the default value for this field when the record is updated. 

compute is an optional function. If a record is inserted or updated, the 
compute function will be executed and the field will be populated with 
the function result. The record is passed to the compute function as a 
diet, and the diet will not include the current value of that, or any other 
compute field. 

authorize can be used to require access control on the corresponding field, 
for "upload" fields only. It will be discussed more in detail in the context 
of Authentication and Authorization. 

autodelete determines if the corresponding uploaded file should be 
deleted when the record referencing the file is deleted. For "upload" fields 
only. 

represent can be None or can point to a function that takes a field value 
and returns an alternate representation for the field value. Examples: 


db 

. mytable 

. name 

. represent = 

lambda name 

,row: name. 

capitalize 



db. 

. mytable 

.othei 

■_id. represent = lambda 

id, row: 

row 

. myfield 


db. 

.mytable 

, some. 

.uploadfield 

. represent = 

lambda 

vali 

je,row: \ 



A( 'get 

it' , 

_href=URL( ' 

download' , a 

rgs=vali 

Je)) 
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"blob" fields are also special. By default, binary data is encoded in base64 
before being stored into the actual database field, and it is decoded when 
extracted. This has the negative effect of using 25% more storage space than 
necessary in blob fields, but has two advantages. On average it reduces 
the amount of data communicated between web2py and the database server, 
and it makes the communication independent of back-end-specific escaping 
conventions. 

Most attributes of fields and tables can be modified after they are defined: 

1 db. define. table( 'person ' , Field ( 'name' ,default=' ' ) , format= '%(name)s ' ) 

2 db. person. -format = '%(name)s/%(id)s' 

3 db. person. name. default = 'anonymous' 

(notice that attributes of tables are usually prefixed by an underscore to avoid 
conflict with possible field names). 

You can list the tables that have been defined for a given database connection: 

1 >» print db. tables 
z ['person' ] 

You can also list the fields that have been defined for a given table: 

1 >» print db. person. fields 

2 [' id' , 'name' ] 


You can query for the type of a table: 


1 >» print type(db. person) 

2 <class ' gluon.sql. Table' > 


and you can access a table from the DAL connection using: 


>» print type (db[ 'person' ] ) 
<class ' gluon.sql. Table' > 


Similarly you can access fields from their name in multiple equivalent ways: 
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Given a field, you can access the attributes set in its definition: 



including its parent table, tablename, and parent connection: 



A field also has methods. Some of them are used to build queries and we 
will seem them later. A special method of the field object is validate and it 
calls the validators for the field. 

print db. person .name. validate ( 'John' ) 

which returns a tuple (value, error), error is None if the input passes 
validation. 


6.6 Migrations 

def ine_table checks whether or not the corresponding table exists. If it does 
not, it generates the SQL to create it and executes the SQL. If the table does 
exist but differs from the one being defined, it generates the SQL to alter the 
table and executes it. If a field has changed type but not name, it will try 
to convert the data(If you do not want this, you need to redefine the table 
twice, the first time, letting web2py drop the field by removing it, and the 
second time adding the newly defined field so that web2py can create it.). If 
the table exists and matches the current definition, it will leave it alone. In 
all cases it will create the db. person object that represents the table. 
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We refer to this behavior as a "migration". web2py logs all migrations and 
migration attempts in the file "databases/sql.log". 

The first argument of define_ table is always the table name. The other 
unnamed arguments are the fields (Field). The function also takes an optional 
last argument called "migrate" which must be referred to explicitly by name 
as in: 

>» db.define_table( 'person' , Field ( 'name' ) , migrate^' person. table' ) 

The value of migrate is the filename (in the "databases" folder for the 
application) where web2py stores internal migration information for this 
table. These files are very important and should never be removed except 
when the entire database is dropped. In this case, the ".table" files have to be 
removed manually. By default, migrate is set to True. This causes web2py 
to generate the filename from a hash of the connection string. If migrate 
is set to False, the migration is not performed, and web2py assumes that 
the table exists in the datastore and it contains (at least) the fields listed in 
define_table. The best practice is to give an explicit name to the migrate 
table. 

There may not be two tables in the same application with the same migrate 
filename. 

The DAL class also takes a "migrate" argument, which determines the default 
value of migrate for calls to def ine_table. For example, 

>» db = DAL( ' sqlite: //storage. db' , migrate=False) 

will set the default value of migrate to False whenever db.define_ table is 
called without a migrate argument. 

Migrations can be disabled for all tables at the moment of connection: 

db = DAL( . . . ,migrate_enabled=False) 

This is the recommended behaviour when two apps share the same database. 
Only one of the two apps should perform migrations, the other should 
disabled them. 
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6. j Fixing broken migrations 

There are two common problems with migrations and there are ways to 
recover from them. 

One problem is specific with SQLite. SQLite does not enforce column 
types and cannot drop columns. This means that if you have a column 
of type string and you remove it, it is not really removed. If you add 
the column again with a different type (for example datetime) you end up 
with a datetime column that contains strings (junk for practical purposes). 
web2py does not complain about this because it does not know what is in 
the database, until it tries to retrieve records and fails. 

If web2py returns an error in the gluon.sql. parse function when selecting 
records, this is the problem: corrupted data in a column because of the above 
issue. 

The solution consists in updating all records of the table and updating the 
values in the column in question with None. 

The other problem is more generic but typical with MySQL. MySQL does not 
allow more than one ALTER TABLE in a transaction. This means that web2py 
must break complex transactions into smaller ones (one ALTER TABLE at the 
time) and commit one piece at the time. It is therefore possible that part of a 
complex transaction gets committed and one part fails, leaving web2py in a 
corrupted state. Why would part of a transaction fail? Because, for example, 
it involves altering a table and converting a string column into a datetime 
column, web2py tries to convert the data, but the data cannot be converted. 
What happens to web2py? It gets confused about what exactly is the table 
structure actually stored in the database. 

The solution consists of disabling migrations for all tables and enabling fake 

migrations: 

db.define_table( .... ,migrate=False,fake_migrate=True) 

This will rebuild web2py metadata about the table according to the table 
definition. Try multiple table definitions to see which one works (the one 
before the failed migration and the one after the failed migration). Once 
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successful remove the fake_migrate=True attribute. 

Before attempting to fix migration problems it is prudent to make a copy of 
"apprications/yourapp/databases/*. table" files. 

Migration problems can also be fixed for all tables at once: 
1 db = DAL( . . . , fake_migrate_all=True) 

Although if this fails, it will not help in narrowing down the problem. 

6.8 insert 

Given a table, you can insert records 

1 >» db. person .insert(name=",4Zex" ) 

2 1 

3 >» db. person .insert(name="Bo£>") 

4 2 

Insert returns the unique "id" value of each record inserted. 

You can truncate the table, i.e., delete all records and reset the counter of the 
id. 
] >» db. person. truncate( ) 

Now, if you insert a record again, the counter starts again at 1 (this is back- 
end specific and does not apply to Google NoSQL): 

1 >» db. person .insert(name="^!ex") 

2 1 

web2py also provides a bulk_insert method 

1 >» db. person . bulk_insert( [{ 'name' : 'Alex' }, {' name' : 'John'} , {' name' :' Tim'}]) 

2 [3,4,5] 

It takes a list of dictionaries of fields to be inserted and performs multiple 
inserts at once. It returns the IDs of the inserted records. On the supported 
relational databases there is no advantage in using this function as opposed 
to looping and performing individual inserts but on Google App Engine 
NoSQL, there is a major speed advantage. 
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6.9 commit and rollback 

No create, drop, insert, truncate, delete, or update operation is actually 
committed until you issue the commit command 


>» db. commit ( ) 

To check it let's 

insert a new 

record: 


>» db. person .inse 
2 

:rt(name=" 

Bob" 

) 


and roll back, i.e., ignore 

•all 

operations since 

the last commit: 

>» db. rollback! ) 


If you now insert again, the counter will again be set to 2, since the previous 
insert was rolled back. 

1 >» db. person . in sert(name=" Bob" ) 

2 2 

Code in models, views and controllers is enclosed in web2py code that looks 
like this: 



There is no need to ever call commit or rollback explicitly in web2py unless 
one needs more granular control. 


6.20 Raw sql 
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6.10.1 Timing queries 

All queries are automatically timed by web2py. The variable db. -timings is 
a list of tuples. Each tuple contains the raw SQL query as passed to the 
database driver and the time it took to execute in seconds. This variable can 
be displayed in views using the toolbar: 

{{= response. toolbar ( ) }} 


6.10.2 executesql 

The DAL allows you to explicitly issue SQL statements. 

>» print db. executesql ( 'SELECT * FROM person;') 
[(1, u' Massimo'), (2, u' Massimo')] 

In this case, the return values are not parsed or transformed by the DAL, and 
the format depends on the specific database driver. This usage with selects is 
normally not needed, but it is more common with indexes, executesql takes 
two optional arguments: placeholders and as_dict placeholders is an optional 
sequence of values to be substituted in or, if supported by the DB driver, a 
dictionary with keys matching named placeholders in your SQL. 

If as_dict is set to True, and the results cursor returned by the DB driver will 
be converted to a sequence of dictionaries keyed with the db field names. 
Results returned with as_dict = True are the same as those returned when 
applying .as_list() to a normal select. 

[{fieldl: valuel, field2: value2}, {fieldl: valuelb, field2: value2b}] 


6.IO.3 -lastsql 

Whether SQL was executed manually using executesql or was SQL generated 
by the DAL, you can always find the SQL code in db._lastsql. This is useful 
for debugging purposes: 


1 >» rows = db( ). select(db. person. ALL) 

2 >» print db._lastsql 

3 SELECT person. id, person. name FROM person; 
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webzpy never generates queries using the "*" operator, webzpy is always 
explicit when selecting fields. 

6.11 drop 

Finally, you can drop tables and all data will be lost: 
1 >» db. person .drop( ) 

6.22 Indexes 

Currently the DAL API does not provide a command to create indexes on 
tables, but this can be done using the executesql command. This is because 
the existence of indexes can make migrations complex, and it is better to deal 
with them explicitly. Indexes may be needed for those fields that are used in 
recurrent queries. 

Here is an example of how to create an index using SQL in SQLite: 

1 >» db = DAL( ' sqlite: //storage. db' ) 

2 >» db.define_table( 'person ' , Field (' name ') ) 

3 »> db. executesql ('CREATE INDEX IF NOT EXISTS myidx ON person (name); ') 

Other database dialects have very similar syntaxes but may not support the 
optional "IF NOT EXISTS" directive. 

6.23 Legacy databases and keyed tables 

web2py can connect to legacy databases under some conditions. 
The easiest way is when these conditions are met: 

• Each table must have a unique auto-increment integer field called "id" 

• Records must be referenced exclusively using the "id" field. 

When accessing an existing table, i.e., a table not created by webzpy in the 
current application, always set migrate=False. 
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If the legacy table has an auto-increment integer field but it is not called "id", 
web2py can still access it but the table definition must contain explicitly as 
Field ('....',' id ' ) where... is the name of the auto-increment integer field. 

Finally if the legacy table uses a primary key that is not an auto-increment id 
field it is possible to use a "keyed table", for example: 

1 db.define_table( 'account' , 

z Field ( ' accnum ' , ' integer ' ) , 

3 Field ( ' acctype' ) , 

4 Field ( ' accdesc' ) , 

5 prima rykey=[ 'accnum' , 'acctype' ] , 

6 migrate=False) 

• primary key is a list of the field names that make up the primary key. 

• All primarykey fields have a NOT NULL set even if not specified. 

• Keyed table can only refer are to other keyed tables. 

• Referenceing fields must use the reference tablename. fieldname format. 

• The update_ record function is not available for Rows of keyed tables. 

Note that currently this is only available for DB2, MS-SQL, Ingres and 
Informix, but others can be easily added. 

At the time of writing, we cannot guarantee that the primarykey attribute 
works with every existing legacy table and every supported database 
backend. For simplicity, we recommend, if possible, creating a database view 
that has an auto-increment id field. 


6.24 Distributed transaction 

At the time of writing this feature is only supported by PostgreSQL, MySQL 
and Firebird, since they expose API for two-phase commits. 

Assuming you have two (or more) connections to distinct PostgreSQL 
databases, for example: 

1 db_a = DAL( 'postgres. ;//...' ) 

2 db_b = DAL (■ postgres. ://... ') 
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In your models or controllers, you can commit them concurrently with: 
DAL. distributed- transaction. commit (db_a, db_b) 

On failure, this function rolls back and raises an Exception. 

In controllers, when one action returns, if you have two distinct connections 
and you do not call the above function, web2py commits them separately. 
This means there is a possibility that one of the commits succeeds and one 
fails. The distributed transaction prevents this from happening. 


6.25 Manual uploads 

Consider the following model: 

1 >» db.define_table( 'myfile' : Field( 'image' , 'upload' ) ) 

Normally an insert is handled automatically via a SQLFORM or a crud form 
(which is a SQLFORM) but occasionally you already have the file on the 
filesystem and want to upload it programmatically. This can be done in this 
way: 

1 >» stream = open (filename, ' rb' ) 

2 >» db.myfile.insert(image=db. myfile. image. store (st ream, filename) ) 

The store method of the upload field object takes a file stream and a filename. 
It uses the filename to determine the extension (type) of the file, creates a 
new temp name for the file (according to web2py upload mechanism) and 
loads the file content in this new temp file (under the uploads folder unless 
specified otherwise). It returns the new temp name, which is then stored in 
the image field of the db. myfile table. 

The opposite of .store is . retrieve: 

1 >» row = db(db. myfile) .select () .first( ) 

2 >» (filename, stream) = db. myfile. image, retrieve! row. image) 

3 >» import shutil 

4 >» shutil . copyfileobj (stream, open(filename, 'vib' ) ) 
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6.26 Query, Set, Rows 

Let's consider again the table defined (and dropped) previously and insert 
three records: 



You can store the table in a variable. For example, with variable person, you 

could do: 

>» person = db. person 

You can also store a field in a variable such as name. For example, you could 

also do: 

>» name = person. name 

You can even build a query (using operators like ==, !=, <, >, <=, >=, like, 
belongs) and store the query in a variable q such as in: 
>» q = name==',4lex' 

When you call db with a query, you define a set of records. You can store it 
in a variable s and write: 
>» s = db(q) 

Notice that no database query has been performed so far. DAL + Query 
simply define a set of records in this db that match the query. web2py 
determines from the query which table (or tables) are involved and, in fact, 
there is no need to specify that. 


6.27 select 

Given a Set, s, you can fetch the records with the command select: 
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1 >» rows = s. select () 

It returns an iterable object of class gluon.sql.Rows whose elements are Row 
objects, gluon . sql . Row objects act like dictionaries, but their elements can also 
be accessed as attributes, like gluon. storage. Storage. The former differ from 
the latter because its values are readonly. 

The Rows object allows looping over the result of the select and printing the 
selected field values for each row: 

] >» for row in rows: 

z print row. id, row. name 

3 1 Alex 

You can do all the steps in one statement: 

1 >» for row in db(db. person. name=='/Uex' ) .select () : 
z print row. name 

3 Alex 


The select command can take arguments. All unnamed arguments are 
interpreted as the names of the fields that you want to fetch. For example, 
you can be explicit on fetching field "id" and field "name": 



The table attribute ALL allows you to specify all fields: 

1 >» for row in db( ) .select (db. person. ALL) : 

2 print row. name 

3 Alex 

4 Bob 

5 Carl 

Notice that there is no query string passed to db. web2py understands that 
if you want all fields of the table person without additional information then 
you want all records of the table person. 

An equivalent alternative syntax is the following: 
1 >» for row in db(db. person. id > 0).select(): 
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2 

3 

Alex 

print row. name 

4 

Bob 


5 

Carl 



and \veb2py understands that if you ask for all records of the table person 
(id > o) without additional information, then you want all the fields of table 
person. 

Given one row 

row = rows [0] 

you can extract its values using multiple equivalent expressions: 



The latter syntax is particularly handy when selecting en expression instead 
of a column. We will show this later. 

You can also do 

rows. compact = False 

to disable the notation 

row[i] .name 

and enable, instead, the less compact notation: 

row[i] .person . name 

Yes this is unusual and not rarely needed. 

6.17.1 Shortcuts 

The DAL supports various code-simplifying shortcuts. In particular: 

myrecord = db.mytable[id] 
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returns the record with the given id if it exists. If the id does not exist, it 
returns None. The above statement is equivalent to 

myrecord = db(db. mytable. id==id) . select( ) . first( ) 

You can delete records by id: 

del db.mytable[id] 

and this is equivalent to 

db(db.mytable.id==id) . delete ( ) 

and deletes the record with the given id, if it exists. 

You can insert records: 

db.mytable[0] = dict(myfie\d=' somevalue' ) 

It is equivalent to 

db.mytable. insert (my field= 'someva Zue' ) 

and it creates a new record with field values specified by the dictionary on 
the right hand side. 

You can update records: 

db.mytable [id] = dict(myfield='somei/alue' ) 

which is equivalent to 

db (db . mytable . id==id ) . update (myf ield= ' somevalue ' ) 

and it updates an existing record with field values specified by the dictionary 
on the right hand side. 

6.17.2 Fetching a Rou 

Yet another convenient syntax is the following: 


record = 

= db 

.mytable(id) 



record = 

= db 

,mytable(db. 

mytable. 

.id==id) 

record = 

= db 

,mytable(id, 

myfield= 

= ' somevalue' ) 
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Apparently similar to db.mytable[id] the above syntax is more flexible and 
safer. First of all it checks whether id is an int (or str(id) is an int) and 
returns None if not (it never raises an exception). It also allows to specify 
multiple conditions that the record must meet. If they are not met, it also 
returns None. 


6.27.3 Recursive selects 

Consider the previous table person and a new table "dog" referencing a 

"person": 

>» db.define_table( 'dog' , Field (' name ') , Field ( 'owner' ,db. person ) ) 

and a simple select from this table: 
>» dogs = db(db. dog) .select( ) 

which is equivalent to 

>» dogs = db(db. dog. _id>0) . select ( ) 

where ._id is a reference to the primary key of the table. Normally db.dog._id 
is the same as db . dog . id and we will assume that in most of this book. 

For each Row of dogs it is possible to fetch not just fields from the selected 
table (dog) but also from linked tables (recursively): 
>» for dog in dogs: print dog. name, dog. owner. name 

Here dog. owner. name requires one database select for each dog in dogs and it 
is therefore inefficient. We suggest using joins whenever possible instead of 
recursive selects, nevertheless this is convenient and practical when accessing 
individual records. 

You can also do it backwards, by selecting the dogs referenced by a person: 

person = db. person(id) 

for dog in person. dog. select (orderby=db. dog. name) : 
print person. name, 'owns', dog. name 

In this last expressions person .dog is a shortcut for 

db(db.dog.owner==person.id) 
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i.e. the Set of dogs referenced by the current person. This syntax breaks down 
if the referencing table has multiple references to the referenced table. In this 
case one needs to be more explicit and use a full Query. 


6.17.4. Serializing Rows in views 

Given the following action containing a query 

def index() 

return dict(rows = db(query) . select) ) ) 

The result of a select can be displayed in a view with the following syntax: 

{{extend ' layout. html ' }} 

<hl>Records</hl> 

{{=rows}} 

Which is equivalent to: 

{{extend 'layout.html'}} 

<hl>Records</hl> 

{{=SQLTABLE(rows)}} 

SQLTABLE converts the rows into an HTML table with a header containing the 
column names and one row per record. The rows are marked as alternating 
class "even" and class "odd". Under the hood, Rows is first converted into 
a SQLTABLE object (not to be confused with Table) and then serialized. 
The values extracted from the database are also formatted by the validators 
associated to the field and then escaped. (Note: Using a db in this way in a 
view is usually not considered good MVC practice.) 

Yet it is possible and sometimes convenient to call SQLTABLE explicitly. 

The SQLTABLE constructor takes the following optional arguments: 

• linkto the URL or an action to be used to link reference fields (default to 
None) 

• upload the URL or the download action to allow downloading of uploaded 
files (default to None) 
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• headers a dictionary mapping field names to their labels to be used as 
headers (default to {}). It can also be an instruction. Currently we support 
headers=' fieldname: capitalize'. 

• truncate the number of characters for truncating long values in the table 
(default is 16) 

• columns the list of fieldnames to be shown as columns (in 
tablename. fieldname format). 

Those not listed are not displayed (defaults to all). 

• **attributes generic helper attributes to be passed to the most external 
TABLE object. 

Here is an example: 

1 {{extend ' layout.html ' }} 

2 <hl>Records</hl> 

3 {{=SQLTABLE(rows, 

4 headers^' fieldname: capitalize' , 

5 truncate=100, 

6 upload=URL( 'download' ) ) 

7 }} 

SQLTABLE is useful but there are types when one needs more. SQLFORM.grid 
is an extension of SQLTABLE that creates a table with search features and 
pagination, as well as ability to open detailed records, create, edit and delete 
records. SQLFORM.smartgrid is a further generalization that allows all of the 
above but also creates buttons to access referencing records. 

Here is an example of usage of SQLFORM.grid: 

1 def index( ) : 

2 return diet (grid=SQLFORM.grid(query) ) 

and the corresponding view: 

1 {{extend 'layout.html'}} 

2 {{=grid}} 

SQLFORM.grid and SQLFORM.smartgrid should be preferrable to SQLTABLE 
because they are more powerful although higher level and therefore more 
constraining. They will be explained in more detail in chapter 8. 
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6.IJ.5 orderby, groupby, limitby, distinct 

The select command takes five optional arguments: orderby, groupby, 
limitby, left and cache. Here we discuss the first three. 

You can fetch the records sorted by name: 



You can fetch the records sorted by name in reverse order (notice the tilde): 


1 

>» for 

row in db( ) .select( 

3 


db. person. ALL, orderby=-db. person. name) : 
print row. name 

4 

Carl 


5 

Bob 


6 

Alex 



You can have the fetched records appear in random order: 


>» for row in db().select( 


db. person. ALL, orderby= 

'<random>' ) : 

print row. name 


Carl 


Alex 


Bob 



The use o/orderby='<random>' is not supported on Google NoSQL. However, 
in this situation and likewise in many others where built-ins are insufficient, 
imports can be used: 

i import random 

2 rows=db( ...) .select!) .sort (lambda row: random. randomf ) ) 

And you can sort the records according to multiple fields by concatenating 
them with a " I ": 
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Bob 

Alex 

Using groupby together with orderby, you can group records with the same 
value for the specified field (this is back-end specific, and is not on the Google 
NoSQL): 



With the argument distinct=True, you can specify that you only want to select 
distinct records. This has the same effect as grouping using all specified fields 
except that it does not require sorting. When using distinct it is important 
not to select ALL fields, and in particular not to select the "id" field, else all 
records will always be distinct. 

Here is an example: 



Notice that distinct can also be an expression for example: 


1 

>» for row in db( ) .select (db. person. name, distinct=db. person. name) : 

2 

print row. name 

3 

Alex 

4 

Bob 

5 

Carl 


With limitby, you can select a subset of the records (in this case, the first two 
starting at zero): 
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6.1J.6 Logical operators 

Queries can be combined using the binary AND operator "&": 


] >» rows = db( (db. person. r\ame==' Alex' ) & (db. person. id>3) ) 

.select!) 

2 >» for row in rows: print row. id, row. name 


3 4 Alex 



and the binary OR operator " | ": 


1 >» rows = db( (db. person, name--' Alex' ) | (db. person. id>3) ). select ( ) 

2 >» for row in rows: print row. id, row. name 
, 1 Alex 


You can negate a query (or sub-query) with the " ! =" binary operator: 



or by explicit negation with the " " unary operator: 



Due to Python restrictions in overloading "and" and "or" operators, these 
cannot be used in forming queries. The binary operators must be used 
instead. 

It is also possible to build queries using in-place logical operators: 


>» 

query 

= 

db.| 

aerson.i 

iame!= 

'Alex' 

>» 

query 

&= 

= db 

.person 

.id>3 


>» 

query 

1 = 

= db 

.person 

. name= 

=' John' 


6.1J.J count, isempty, delete, update 

You can count records in a set: 
>» print db(db. person. id > 8). count () 


2 3 
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Notice that count takes an optional distinct argument which defaults to 
False, and it works very much like the same argument for select. 

Sometimes you may need to check is a table is empty. A more efficient way 
than counting is using the isempty method: 

>» print db(db. person. id > 0) .isempty ( ) 
False 

or equivalently: 

>» print db(db. person) .isempty! ) 
False 

You can delete records in a set: 
>» db(db. person. id > 3).delete() 

And you can update all records in a set by passing named arguments 
corresponding to the fields that need to be updated: 

>» db(db. person. id > 3) . update) name= 'Ken' ) 


6.1J.8 Expressions 

The value assigned an update statement can be an expression. For example 
consider this model 



The values used in queries can also be expressions 
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6.IJ.9 update_ record 

weD2py also allows updating a single record that is already in memory using 
update_record 

>» row = db(db. person. id==2) . select) ) .first ( ) 
>» row. update. record (name=' Curt ' ) 

update_ record should not be confused with 
>» row. update (name=' Curt ' ) 

because for a single row, the method update updates the row object but not 
the database record, as in the case of update_ record. 

It is also possible to change the attributes of a row (one at the time) and then 
call update_record() without arguments to save the changes: 


1 

>» 

row 

= db(db 

.person. 

id 

> 2). 

select ().first() 

2 

>» 

row. 

name = 

■Curf 




3 

>» 

row. 

update. 

record ( ) 

# 

saves 

: above change 


6.1J.10 first and last 

Given a Rows object containing records: 


>» 

rows 

= db(q 

uery) 

. select () 

»> 

first 

_row = 

rows 

.first() 

>» 

last. 

row = 

rows . 

last() 


are equivalent to 


1 >» first_row = rows[0] if len(rows)>0 else None 

2 >» last_row = rows[-l] if len(rows)>0 else None 


6.I7.II as_dict and as_list 

A Row object can be serialized into a regular dictionary using the as_dict() 
method and a Rows object can be serialized into a list of dictionaries using 
the as_list( ) method. Here are some examples: 
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>» 

rows 

= db(query) 

. select () 




>» 

rows_ 

list 

= rows 

.as_list() 




>» 

first 

_row_ 

.diet = 

rows .firs 

t() 

.as. 

-dict() 


These methods are convenient for passing Rows to generic views and or to 
store Rows in sessions (since Rows objects themselves cannot be serialized 
since contain a reference to an open DB connection): 


>» rows = db(query) .select( ) 

>» session, rows = rows # not allowed! 

>» session, rows = rows.as_list ( ) # allowed! 


6.1J.12 find, exclude, sort 


There are times when one needs to perform two selects and one contains a 
subset of a previous select. In this case it is pointless to access the database 
again. The find, exclude and sort objects allow you to manipulate a Rows 
objects and generate another one without accessing the database. More 
specifically: 

• find returns a new set of Rows filtered by a condition and leaves the 
original unchanged. 

• exclude returns a new set of Rows filtered by a condition and removes 
them from the original Rows. 

• so rt returns a new set of Rows sorted by a condition and leaves the original 
unchanged. 

All these methods take a single argument, a function that acts on each 
individual row. 

Here is an example of usage: 


1 

>» 

db.define_table( 'person' , Field ( 'name' ) ) 


2 

>» 

db. person. insert(name=' John' ) 


3 

»> 

db. person . insert (name=' Max' ) 


4 

»> 

db. person . insert (name='/<H ex' ) 


5 

»> 

rows = db(db. person) . select ( ) 


6 

»> 

for row in rows.find(lambda row: row. name [0]== 

'«'): 

7 


print row. name 


8 

Max 
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They can be combined: 


1 >» rows = db(db. person) .select ( ) 

2 >» rows = rows.find( 

3 lambda row: 'x' in row. name ) .sort ( 

4 lambda row: row. name) 

5 >» for row in rows: 

6 print row. name 
- Alex 

8 Max 


6.18 Other methods 


6.18.I update_or_insert 

Some times you need to perform an insert only if there is no record with the 

same values as those being inserted. This can be done with 

db.define_table( 'person ' , Field ( 'name' ) , Field ( 'birthplace' ) ) 
db. person. update. or_insert (name=' John ' , birthplace=' Chicago ' ) 

The record will be inserted only of there is no other user called John born in 
Chicago. 

You can specify which values to use as a key to determine if the record exists. 

For example: 

db. person . update_or_insert (db . person . name==' John ' , 
name= ' John ' , bi rthplace= ' Chicago ' ) 

and if there is John his birthplace will be updated else a new record will be 
created. 
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6.18.2 validate_and_insert, validate_and_update 

The function 

ret = db.mytable.validate_and_insert(field=' value' ) 

works very much like 

id = db. mytable. insert ( f ield= ' value' ) 

except that it calls the validators for the fields before performing the insert 
and bails out if the validation does not pass. If validation does not pass the 
errors can be found in ret. error. If it passes, the id of the new record is in 
ret. id. Mind that normally validation is done by the form processing logic 
so this function is rarely needed. 

Similarly 

ret = db(query) . validate_and_update(field=' value' ) 

works very much the same as 

num = db(query) . update (field=' value' ) 

except that it calls the validators for the fields before performing the update. 
Notice that it only works if query involves a single table. The number of 
updated records can be found in res . updated and errors will be ret .errors. 

6.18.3 smart_query (experimental) 

There are times when you need to parse a query using natural language such 

as 

name contain m and age greater than 18 

The DAL provides a method to parse this type of queries: 

search = 'name contain m and age greater than 18' 
rows = db.smart_query( [db. person] , search) . select ( ) 

The first argument must be a list of tables or fields that should be allowed 
in the search. It raises a RuntimeError if the search string is invalid. This 
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functionality can be used to build RESTful interfaces (see chapter 10) and it 
is used internally by the SQLFORM.grid and SQLFORM.smartgrid. 

In the smartquery search string, a field can be identified by fieldname only 
and or by tablename. fieldname. Strings may be delimited by double quotes 
if they contain spaces. 


6.29 Computed fields 

DAL fields may have a compute attribute. This must be a function (or lambda) 
that takes a Row object and returns a value for the field. When a new record 
is modified, including both insertions and updates, if a value for the field is 
not provided, web2py tries to compute from the other field values using the 
compute function. Here is an example: 



Notice that the computed value is stored in the db and it is not computed 
on retrieval, as in the case of virtual fields, described later. Two typical 
applications of computed fields are: 

• in wiki applications, to store the processed input wiki text as HTML, to 
avoid re-processing on every request 

• for searching, to compute normalized values for a field, to be used for 
searching. 


6.20 Virtual fields 
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6.20.1 Old style virtual fields 

Virtual fields are also computed fields (as in the previous subsection) but 
they differ from those because they are virtual in the sense that they are not 
stored in the db and they are computed each time records are extracted from 
the database. They can be used to simplify the user's code without using 
additional storage but they cannot be used for searching. 

In order to define one or more virtual fields, you have to define a container 

class, instantiate it and link it to a table or to a select. For example, consider 

the following table: 

1 >» db.define_table( 'item' , 

z Field( 'unit-price' , 'double' ) , 

3 Field ( 'quantity ', 'integer' ) , 

One can define a total_price virtual field as 

1 >» class MyVirtualFields(object) : 

2 def total_price(self ) : 

3 return self .item. unit_price*self . item. quantity 

4 >» db.item. virtualfields .append(MyVirtualFields( ) ) 

Notice that each method of the class that takes a single argument (self) is 
a new virtual field, self refers to each one row of the select. Field values 
are referred by full path as in self .item. unit_p rice. The table is linked to the 
virtual fields by appending an instance of the class to the table's virtualfields 
attribute. 

Virtual fields can also access recursive fields as in 

1 >» db.define_table( 'item' , 

2 Field( 'unit-price' , 'double' ) ) 

3 >» db. def ine_table ( ' order_item' , 

4 Field ( 'item' , db.item) , 

5 Field ( 'quantity ', 'integer' ) ) 

6 >» class MyVirtualFields(object) : 

7 def total_price(self ) : 

s return self .order_item. item. unit_price \ 

i) * self .order_item. quantity 

10 >» db. order_item. virtualfields .append(MyVirtualFields( ) ) 

Notice the recursive field access self . order_item. item. unit_price where self 
is the looping record. 
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They can also act on the result of a JOIN 



Notice how in this case the syntax is different. The virtual field accesses 
both self .item. unit_p rice and self . order_item. quantity which belong to the 
join select. The virtual field is attached to the rows of the table using the 
setvirtualfields method of the rows object. This method takes an arbitrary 
number of named arguments and can be used to set multiple virtual fields, 
defined in multiple classes, and attach them to multiple tables: 

1 >» class MyVirtualFieldsl(object) : 

2 def discounted_unit_price(self ) : 

3 return self .item. unit_price*0. 90 

4 >» class MyVirtualFields2(object) : 

5 def total_price(self ) : 

6 return self .item. unit_price \ 

7 * self .order_item. quantity 
s def discounted_total_price(self ) : 

i) return self .item. discounted_unit_price \ 

10 * self .order_item. quantity 

n >» rows.setvirtualfields( 

12 item=HyVirtualFieldsl( ) , 

13 order_item=MyVirtualFields2( ) ) 

14 >» for row in rows: 

15 print row.order_item.discounted_total_price 

Virtual fields can be lazy; all they need to do is return a function and access 
it by calling the function: 

1 >» db. def ine_table( 'item' , 

2 Field( 'unit-price' , 'double' ) , 

3 Field ( 'quantity' , 'integer' ) , 

4 >» class MyVirtualFields(object) : 

5 def lazy_total_price(self ) : 

6 def lazy(self=self ) : 
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7 return self .item. unit_price \ 

s * self .item. quantity 

i) return lazy 

[o >» db.item.virtualfields .append(MyVirtualFields( ) ) 

11 >» for item in db(db. item) . select () : 

12 print item.lazy_total_price( ) 

or shorter using a lambda function: 

] >» class MyVirtualFields(object) : 
z def lazy_total_price(self ) : 

3 return lambda self=self: self . item. unit_price \ 

4 * self .item. quantity 


6.20.2 New style virtual fields (experimental) 

web2py provides a new and easier way to define virtual fields and lazy 
virtual fields. This section is marked experimental because they APIs may 
still change a little from what is described here. 

Here we will consider the same example as in the previous subsection. In 
particular we consider the following model: 

>» db.define_table( 'item' , 

Field ( ' unit-price' , 'double' ) , 
Field ( 'quantity' , 'integer' ) , 

One can define a total_price virtual field as 

>» db.item.total_price = Field. Virtual(lambda row: row. unit_price*row. quantity) 

i.e. by simply defining a new field total_price to be a Field. Virtual. The 
only argument of the constructor is a function that takes a row and returns 
the computed values. 

A virtual field defined as the one above is automatically computed for all 

records when the records are selected: 

>» for row in db(db. item) . select () : print row.total_price 

It is also possible to define lazy virtual fields which are calculated on- 
demand, when called. For example: 


THE DATABASE ABSTRACTION LAYER 285 


>» db.item. total_price = Field. Lazy (lambda row, discount=0.0: \ 
row. unit_p rice* row. quantity* (1.0 -discount/ 100) ) 

In this case row.total_price is not a value but a function. The function takes 
the same arguments as the function passed to the Lazy constructor except for 
row which is implicit (think of it as self for rows objects). 

The lazy field in the example above allows one to compute the total price for 

each item: 

>» for row in db(db. item) . select () : print row.total_price( ) 

And it also allows to pass an optional discount percentage (15%): 
>» for row in db(db.item) .select( ) : print row.total_price(15) 

Mind that virtual fields do not have the same attributes as the other fields 
(default, readable, requires, etc) and they do not appear in the list of 
db. table, fields and are not visualized by default in tables (TABLE) and grids 
(SQLFORM.grid,SQLFORM.smartgrid). 


6.21 One to many relation 

To illustrate how to implement one to many relations with the web2py DAL, 
define another table "dog" that refers to the table "person" which we redefine 
here: 



Table "dog" has two fields, the name of the dog and the owner of the dog. 
When a field type is another table, it is intended that the field reference the 
other table by its id. In fact, you can print the actual type value and get: 


>» print db. dog. owner. type 
reference person 
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Now, insert three dogs, two owned by Alex and one by Bob: 

1 >» db. dog. insert (name=' Skipper' , owner=l) 

2 1 

3 >» db. dog. insert (name= 'Snoopy' , owner=l) 

4 2 

5 >» db.dog.insert(name='Puppy' , owner=2) 

6 3 

You can select as you did for any other table: 


>» for 

row in 
print 

db(db.dog. 
row. name 

owner= 

=1) 

. select () : 

Skipper 






Snoopy 







Because a dog has a reference to a person, a person can have many dogs, so 
a record of table person now acquires a new attribute dog, which is a Set, 
that defines the dogs of that person. This allows looping over all persons and 
fetching their dogs easily: 



6.21.1 Inner joins 

Another way to achieve a similar result is by using a join, specifically an 
INNER JOIN. web2py performs joins automatically and transparently when 
the query links two or more tables as in the following example: 
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Observe that weD2py did a join, so the rows now contain two records, one 
from each table, linked together. Because the two records may have fields 
with conflicting names, you need to specify the table when extracting a field 
value from a row. This means that while before you could do: 

1 row. name 

and it was obvious whether this was the name of a person or a dog, in the 
result of a join you have to be more explicit and say: 
1 row. person. name 

or: 
1 row. dog. name 

There is an alterantive syntax for INNER JOINS: 

1 >» rows = db(db. person) .select (join=db. dog. on(db. person. id==db. dog. owner) ) 

2 >» for row in rows: 

3 print row. person. name, 'has', row. dog. name 

4 Alex has Skipper 

5 Alex has Snoopy 


6 Bob has Puppy 


While the output is the same, the generated SQL in the two cases can be 
different. The latter syntax removes possible ambiguities when the same 
table is joined twice and aliased: 


>» db.define_table( 'dog' , 




Field ( 'name' ) , 




Field ( ' ownerl ' ,db. person) , 




Field ( ' owner2' ,db. person) ) 




>» rows = db(db. person) . select ( 




join=[db. person. wit h_alias( 'ownerl ' \ 

).on(db. 

. person. id==db 

.dog. ownerl) . 

db. person. wit h_alias( ' owner2' \ 

).on(db. 

, person. id==db 

. dog.owner2) ] ) 


The value of j oin can be list of db . table . on ( . . . ) to join. 

6.21.2 Left outer join 

Notice that Carl did not appear in the list above because he has no dogs. If 
you intend to select on persons (whether they have dogs or not) and their 
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dogs (if they have any), then you need to perform a LEFT OUTER JOIN. This 
is done using the argument "left" of the select command. Here is an example: 



where: 

left = db.dog . on( . . . ) 

does the left join query. Here the argument of db.dog. on is the condition 
required for the join (the same used above for the inner join). In the case of a 
left join, it is necessary to be explicit about which fields to select. 

Multiple left joins can be combined by passing a list or tuple of 
db . mytable . on ( . . . ) to the left attribute. 


6.21.3 Grouping and counting 

When doing joins, sometimes you want to group rows according to certain 
criteria and count them. For example, count the number of dogs owned by 
every person. web2py allows this as well. First, you need a count operator. 
Second, you want to join the person table with the dog table by owner. Third, 
you want to select all rows (person + dog), group them by person, and count 
them while grouping: 


1 

>» count = db. person. id. count ( ) 

2 

>» for row in db(db. person. id==db. dog. owner) . select) 

3 

db. person. name, count, g roupby=db. person. name ) : 

4 

print row. person. name, row[count] 

5 

Alex 2 

6 

Bob 1 


Notice the count operator (which is built-in) is used as a field. The only issue 
here is in how to retrieve the information. Each row clearly contains a person 
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and the count, but the count is not a field of a person nor is it a table. So 
where does it go? It goes into the storage object representing the record with 
a key equal to the query expression itself. 

6.22 Many to many 

In the previous examples, we allowed a dog to have one owner but one 
person could have many dogs. What if Skipper was owned by Alex and Curt? 
This requires a many-to-many relation, and it is realized via an intermediate 
table that links a person to a dog via an ownership relation. 

Here is how to do it: 



the existing ownership relationship can now be rewritten as: 


>» db. 

.ownership. 

,insert(person=l, 

dog=l) 

# Alex owns Skipper 

>» db. 

.ownership. 

,insert(person=l, 

dog=2) 

# Alex owns Snoopy 

>» db. 

.ownership. 

,insert(person=2, 

dog=3) 

# Bob owns Puppy 


Now you can add the new relation that Curt co-owns Skipper: 
>» db. ownership. insert(person=3, dog=l) # Curt owns Skipper too 

Because you now have a three-way relation between tables, it may be 
convenient to define a new set on which to perform operations: 


>» persons_and_dogs = db( 


(db . person . id==db . ownership .person) 

\ 

& (db. dog. id==db. ownership. dog) ) 



Now it is easy to select all persons and their dogs from the new Set: 


1 >» for row in persons_and_dogs. select( ) : 
z print row. person. name, row. dog. name 

3 Alex Skipper 
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4 Alex Snoopy 

5 Bob Puppy 

6 Curt Skipper 

Similarly, you can search for all dogs owned by Alex: 



and all owners of Skipper: 

1 >» for row in persons_and_dogs(db. dog. name— ' Skipper' ) .select () : 

2 print row. person. name 

3 Alex 

4 Curt 

A lighter alternative to Many 2 Many relations is tagging. Tagging is 
discussed in the context of the IS_II\LDB validator. Tagging works even on 
database backends that do not support JOINs like the Google App Engine 
NoSQL. 


6.23 Many to many, ust:<type>, and contains 

web2py provides the following special field types: 

1 list:string 

2 list: integer 

3 list : reference <table> 

They can contain lists of strings, of integers and of references respectively. 

On Google App Engine NoSQL list: string is mapped into 
StringListProperty, the other two are mapped into ListProperty(int). 
On relational databases they all mapped into text fields which contain the 
list of items separated by | . For example [1,2,3] is mapped into 1 1 1 2 1 3 1 . 

For lists of string the items are escaped so that any | in the item is replaced 
by a I I . Anyway this is an internal representation and it is transparent to the 
user. 

You can use list : st ring, for example, in the following way: 
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list : integer works in the same way but the items must be integers. 

As usual the requirements are enforced at the level of forms, not at the level 
of insert. 

For list :<type> fields the contains (value) operator maps into a non trivial 
query that checks for lists containing the value. The contains operator also 
works for regular string and text fields and it maps into a LIKE '%value%'. 

The list : reference and the contains(value) operator are particularly useful 
to de-normalize many-to-many relations. Here is an example: 



Notice that a list : reference tag field get a default constraint 
requires = IS_IN_DB(db, ' tag. id' : db. tag. -format : multiple=True) 

that produces a SELECT/OPTION multiple drop-box in forms. 

Also notice that this field gets a default represent attribute which represents 
the list of references as a comma-separated list of formatted references. This 
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is used in read forms and SQLTABLEs. 

While list: reference has a default validator and a default representation, 
list [integer and list: string do not. So these two need an IS_IN_SET or an 
IS_IN_DB validator if you want to use them informs. 


6.24 Other operators 

web2py has other operators that provide an API to access equivalent SQL 
operators. Let's define another table "log" to store security events, their 
event_time and severity, where the severity is an integer number. 


>» db.define_table( ' log' 

, Field ( 'event' ) , 



Field ( ' event_tiwe' 

, 'datetime' ) , 


Field( ' severity' , 

'integer' ) ) 


As before, insert a few events, a "port scan", an "xss injection" and an 
"unauthorized login". For the sake of the example, you can log events with 
the same event_time but with different severities (1, 2, 3 respectively). 

1 >» import datetime 

2 >» now = datetime. datetime. now( ) 

3 >» print db.log.insert( 

4 event='porr scan', event_time=now, severity=l) 

5 1 

6 >» print db.log.insert( 

7 event='xss injection' , event_time=now : severity=2) 

8 2 

i, >» print db.log.insert( 
10 event=' unauthorized login', event_time=now, severity=3) 

n 3 


6.24.I like, startswith, contains, upper, lower 

Fields have a like operator that you can use to match strings: 


>» 

for 

row in 

db(db 

.log. 

event 

,like( 

' port%' 

)) 

. select) ) : 



print 

row. event 






port 

scan 
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Here "port%" indicates a string starting with "port". The percent sign 
character, "%", is a wild-card character that means "any sequence of 
characters". web2py also provides some shortcuts: 

1 db.mytable. my field. startswith( 'value' ) 

2 db.mytable. my field. contains ( 'value' ) 

which are equivalent respectively to 

1 db.mytable. my field. like( ' value%' ) 

2 db.mytable. my field. like ( '%value%' ) 

Notice that contains has a special meaning for list:<type> fields and it was 
discussed in a previous section. 

The contains method can also be passed a list of values and an optional 
boolean argument all to search for records that contain all values: 

] db. mytable. myfield. contains! [' valuel ',' value2' ] , all=True) 

or any value from the list 

1 db. mytable. myfield. contains! [ 'valuel ',' value2' ] , all=false) 

The upper and lower methods allow you to convert the value of the field to 
upper or lower case, and you can also combine them with the like operator: 

1 >» for row in dbfdb.log. event .upper( ) .like( ' PORT%' )) .select () : 

2 print row. event 
, port scan 


6.24.2 year, month, day, hour, minutes, seconds 

The date and datetime fields have day, month and year methods. The 
datetime and time fields have hour, minutes and seconds methods. Here 
is an example: 
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6.24.3 belongs 

The SQL IN operator is realized via the belongs method which returns true 
when the field value belongs to the specified set (list of tuples): 



The DAL also allows a nested select as the argument of the belongs operator. 
The only caveat is that the nested select has to be a -select, not a select, and 
only one field has to be selected explicitly, the one that defines the set. 



6.24.4 sum ' m i n ' max an d len 

Previously, you have used the count operator to count records. Similarly, you 
can use the sum operator to add (sum) the values of a specific field from a 
group of records. As in the case of count, the result of a sum is retrieved via 
the store object: 


>» 

>» 

6 

sum = 
print 

db.li 
db() 

:>g. severity, 
.select (sum) 

sum( ) 
.firs 

t()[ 

sum] 


You can also use min and max to the mininum and maximum value for the 
selected records 

>» max = db. log. severity. max ( ) 

>» print db( ). select(max) . first () [max] 

3 

. len ( ) computes the length of a string, text or boolean fields. 
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Expressions can be combined to form more complex expressions. For 
example here we are computing the sum of the length of all the severity 
strings in the logs, increased of one: 

1 >» sum = (db. log. severity. len( )+l) .sum() 

2 >» print db( ). select(sum) . first () [sum] 


6.24.5 Substrings 

One can build an expression to refer to a substring. For example, we can 
group dogs whose name starts with the same three characters and select 
only one from each group: 

db(db. dog) . select (dictinct = db.dog.name[ :3] ) 


6.24.6 Default values with coalesce and coalesce_zero 

There are times when you need to pull a value from database but also need 
a default values if the value for a record is set to NULL. In SQL there is a 
keyword, COALESCE, for this. web2py has an equivalent coalesce method: 

>» db.define_table( ' sysuser' , Field ( ' username' ) , Field ( ' fullname' ) ) 

>» db. sysuser. insert(username='max' ,fullname= 'Max Power') 

>» db. sysuser. in sert( use rname=' tim' ,fullname=None) 

print db(db. sysuser) .select (db. sy suser. full name. coalesce (db. sysuser. username) ) 

" COALESCE ( sysuse r . fullname, sysuser. username) " 

Max Power 

tim 

Other times you need to compute a mathematical expression but some fields 
have a value set to None while it should be zero. coalesce_zero comes to the 
rescue by defaulting None to zero in the query: 

>» db.define_table( 'sysuser' , Field ( 'username' ) , Field ( 'points' ) ) 

>» db. sysuser. insert(username='roax' ,points=10) 

»> db. sysuser. insert(username=' tim' ,points=None) 

>» print db(db. sysuser) . select (db. sysuser. point s.coalesce_zero( ) . sum() ) 

" SUM ( COALESCE ( sysuser. points, 0) ) " 

10 
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6.25 Generating raw sql 

Sometimes you need to generate the SQL but not execute it. This is easy 
to do with web2py since every command that performs database IO has an 
equivalent command that does not, and simply returns the SQL that would 
have been executed. These commands have the same names and syntax as 
the functional ones, but they start with an underscore: 

Here is -insert 


2 

>» print db. person. .insert (name='/Uex' ) 
INSERT INTO person(name) VALUES {'Alex'); 


Here is _count 





>» print db(db. person, name— 'Alex' ) ._count( ) 
SELECT count(*) FROM person WHERE person. name=' 

Alex' ; 


Here is -select 




2 

»> print dbfdb.pe 
SELECT person. id, | 

rson.name=='/llex' ) . 
jerson.name FROM pe 

.select () 
rson WHERE 

person. name=' Alex' ; 


Here is _delete 




2 

>» print db(db.pe 
DELETE FROM person 

rson.name=='/llex' ) . 
WHERE person .name= 

-delete () 
'Alex' ; 



And finally, here 

is _update 



2 

»> print dbfdb.pe 
UPDATE person SET 

rson.name==',4lex' ) . 
WHERE person .name= 

_update() 
'Alex' ; 



Moreover you can always use db ._lastsql to return the most recent SQL code, 
whether it was executed manually using executesql or was SQL generated by 
the DAL. 


6.26 Exporting and importing data 
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6.26.2 CSV (one Table at a time) 

When a DALRows object is converted to a string it is automatically serialized 
in CSV: 



You can serialize a single table in CSV and store it in a file "test.csv": 
>» open( ' test. csv' , V ) .write(str(db(db. person. id) .selectO )) 

and you can easily read it back with: 

>» db. person .import_f rom_csv_file(open( ' test, csv' , 'r')) 

When importing, web2py looks for the field names in the CSV header. In this 
example, it finds two columns: "person. id" and "person. name". It ignores the 
"person." prefix, and it ignores the "id" fields. Then all records are appended 
and assigned new ids. Both of these operations can be performed via the 
appadmin web interface. 

6.26.2 CSV (all tables at once) 

In web2py, you can backup /restore an entire database with two commands: 
To export: 


>» db. expo rt_to_csv_file( open ( ' some file, csv' , 

'wb')) 

To import: 

>» db.import_f rom_ csv_ file (open ( ' some file, csv' 

, 'rb')) 


This mechanism can be used even if the importing database is of a different 
type than the exporting database. The data is stored in "somefile.csv" as a 
CSV file where each table starts with one line that indicates the tablename, 
and another line with the fieldnames: 
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TABLE tablename 

fieldl, field2, field3, . . . 

Two tables are separated \r\n\r\n. The file ends with the line 

END 

The file does not include uploaded files if these are not stored in the database. 
In any case it is easy enough to zip the "uploads" folder separately. 

When importing, the new records will be appended to the database if it is not 
empty. In general the new imported records will not have the same record id 
as the original (saved) records but web2py will restore references so they are 
not broken, even if the id values may change. 

If a table contains a field called "uuid", this field will be used to identify 
duplicates. Also, if an imported record has the same "uuid" as an existing 
record, the previous record will be updated. 

6.26.3 CSV and remote database synchronization 
Consider the following model: 

db = DAL( ' sqlite: memory: ' ) 
db.define_table( 'person ' , 

Field ( ' name ' ) , 

fo rmat='% (name )s ' ) 
db . def ine_table ( ' dog ' , 

Field ( 'owner' , db. person), 

Field ( ' name ' ) , 

fo rmat= '%f name J s ' ) 

if not db(db. person) .count () : 

id = db. person. insert (name= "Massimo") 
db. dog. insert (owner=id, name=" Snoopy" ) 

Each record is identified by an ID and referenced by that ID. If you have two 
copies of the database used by distinct web2py installations, the ID is unique 
only within each database and not across the databases. This is a problem 
when merging records from different databases. 

In order to make a record uniquely identifiable across databases, they must: 
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• have a unique id (UUID), 

• have an event_time (to figure out which one is more recent if multiple 
copies), 

• reference the UUID instead of the id. 

This can be achieved without modifying weD2py. Here is what to do: 

1. Change the above model into: 

1 db.define_table( 'person ' , 

2 Field ( 'uuid' , length=64, default=lambda: str(uuid.uuid4( ) ) ) , 

3 Field ( 'modi fied^on ' , 'datetime', default=now) , 

4 Field ( ' name ' ) , 

5 format= ' %(name)s ' ) 

6 

- db.define_table( 'dog' , 

8 Field( 'uuid' , length=64, default=lambda: str(uuid.uuid4( ) ) ) , 

I, Field ( 'modi fied-on ' , 'datetime', default=now) , 
10 Field ( 'owner' , length=64) , 
u Field (' name ') , 
12 fo rmat= '%f name J s ' ) 

13 

14 db. dog. owner. requires = IS_IN_DB(db, 'person, uuid' , '%(name)s' ) 

it if not db(db. person. id) .count () : 

17 id = uuid.uuid4() 

18 db. person .insert(name="Massimo" , uuid=id) 

19 db. dog. insert (owner=id, name=" Snoopy") 

Note, in the above table definitions, the default value for the two 'uuid' fields 
is set to a lambda function, which returns a UUID (converted to a string). 
The lambda function is called once for each record inserted, ensuring that each 
record gets a unique UUID, even if multiple records are inserted in a single 
transaction. 

2. Create a controller action to export the database: 

1 def export ( ) : 

s = StringlO.StringlOO 

3 db.export_to_csv_file(s) 

4 response. headers! 'Content-Type' ] = 'text/csv' 

5 return s.getvalue() 

3. Create a controller action to import a saved copy of the other database and 
sync records: 
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def import_and_sync( ) : 

form = FORM(INPUT(_type='fiIe' , _name=' data' ) , INPUT(_type=' submit ') ) 
if form. process(session=None) .accepted: 

db.import_f rom_csv_file(form.vars .data.file,unique=False) 

# for every table 

for table in db. tables: 

# for every uuid, delete all but the latest 
items = db(db[table]).select(db[table] .id, 
db[table] .uuid, 

orderby=db[table] .modified_on, 
groupby=db[table] .uuid) 
for item in items: 

db( (db[ table] . uuid==item. uuid)&\ 
(db[table] .id !=item.id) ) . delete () 
return diet (form=form) 

Notice that session =N one disables the CSRF protection since this URL is 
intended to be accessed from outside. 

4. Create an index manually to make the search by uuid faster. 

Notice that steps 2 and 3 work for every database model; they are not specific 
for this example. 

Alternatively, you can use XML-RPC to export /import the file. 

If the records reference uploaded files, you also need to export/import the 
content of the uploads folder. Notice that files therein are already labeled by 
UUIDs so you do not need to worry about naming conflicts and references. 


6.26.4 HTML and XML (one Table at a time) 

DALRows objects also have an xml method (like helpers) that serializes it to 

XML/HTML: 

>» rows = db(db. person. id > 0).select() 
>» print rows. xml () 
<table> 
<thead> 
<tr> 

<th>person . id</th> 

<th>person . name</th> 

<th>dog .id</th> 

<th>dog . name</th> 

<th>dog .owner</th> 
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11 

</tr> 

12 

</thead> 

13 

<tbody> 

14 

<tr class="eve/7"> 

15 

<td>l</td> 

16 

<td>Alex</td> 

*7 

<td>l</td> 

18 

<td>Skipper</td> 

19 

<td>l</td> 

20 

</tr> 

22 

</tbody> 

=3 

</table> 


If you need to serialize the DALRows in any other XML format with custom 
tags, you can easily do that using the universal TAG helper and the * notation: 


1 

>» rows = db(db. person. id > 0) . select( ) 

2 

>» print TAG. result (* [TAG. row(*[TAG.field( r[f ] , _name=f) \ 

3 

for f in db. person. fields] ) for r in rows]) 

4 

<result> 

5 

<row> 

6 

<field name="id">l</field> 

7 

<field name=" name ">Alex</field> 

8 

</row> 

9 

</result> 


6.26.5 Data representation 

The export_to_csv_file function accepts a keyword argument named 
represent. When True it will use the columns represent function while 
exporting the data instead of the raw data. 

The function also accepts a keyword argument named colnames that should 
contain a list of column names one wish to export. It defaults to all columns. 

Both export_to_csv_f ile and import_f rom_csv_f ile accept keyword arguments 
that tell the csv parser the format to save /load the files: 

• delimiter: delimiter to separate values (default ',') 

• quotecha r: character to use to quote string values (default to double quotes) 
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• quoting: quote system (default csv.QUOTE_MINIMAL) 
Here is some example usage: 

1 >» import csv 

2 >» db. export_to_csv_fi"le(open( ' /tmp/test. txt' , V), 

3 delimiter^' | ' , 

4 quotechar=' " ' , 

5 quoting=csv.QUOTE_NONNUMERIC) 

Which would render something similar to 
, "hello" 1 35 1 "this is the text description" | "2009-03-03" 

For more information consult the official Python documentation [67] 


6. 2 j Caching selects 

The select method also takes a cache argument, which defaults to None. 
For caching purposes, it should be set to a tuple where the first element is 
the cache model (cache. ram, cache. disk, etc.), and the second element is the 
expiration time in seconds. 

In the following example, you see a controller that caches a select on the 
previously defined db.log table. The actual select fetches data from the back- 
end database no more frequently than once every 60 seconds and stores the 
result in cache. ram. If the next call to this controller occurs in less than 60 
seconds since the last database IO, it simply fetches the previous data from 
cache, ram. 

def cache_db_select( ) : 

logs = db( ) . select(db.log . ALL, cache=( cache. ram, 60)) 
return diet (logs=logs) 

The results of a select are complex, un-pickleable objects; they cannot be stored 
in a session and cannot be cached in any other way than the one explained here. 
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6.28 Self-Reference and aliases 

It is possible to define tables with fields that refer to themselves although the 
usual notation may fail. The following code would be wrong because it uses 
a variable db. person before it is defined: 



The solution consists of using an alternate notation 



In fact db.tablename and "reference tablename" are equivalent field types. 

If the table refers to itself, then it is not possible to perform a JOIN to select a 
person and its parents without use of the SQL "AS" keyword. This is achieved 
in web2py using the with_alias. Here is an example: 



Notice that we have chosen to make a distinction between: 
• "father_id": the field name used in the table "person"; 
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• "father": the alias we want to use for the table referenced by the above 
field; this is communicated to the database; 

• "Father": the variable used by web2py to refer to that alias. 

The difference is subtle, and there is nothing wrong in using the same name 
for the three of them: 

1 db.define_table( 'person ' , 

2 Field ( ' name ' ) , 

3 Field (' father' , 'reference person'), 

4 Field ( 'mother' , 'reference person')) 

5 >» father = db. person. with_alias( ' father' ) 

6 >» mother = db. person. with_alias( 'mother' ) 
- >» db. person . in sert(name=' Massimo' ) 

s 1 

i, >» db. person .insert(name='CIaudia' ) 

10 2 

i] >» db. person .insert(name=' Marco' , father=l, mother=2) 

12 3 

[ 3 >» rows = db( ). select(db. person. name, father, name, mother. name, 

14 left=(father.on(father.id==db.person. father) , 

15 mother.on(mother.id==db.person.mother) ) ) 
it, >» for row in rows: 

17 print row. person. name, row. father. name, row. mother. name 

[8 Massimo None None 

i 9 Claudia None None 

20 Marco Massimo Claudia 

But it is important to have the distinction clear in order to build correct 
queries. 


6.29 Advanced features 

6.29.1 Table inheritance 

It is possible to create a table that contains all the fields from another table. 
It is sufficient to pass the other table in place of a field to def ine_table. For 
example 

1 db. def ine_table( 'person ' , Field (' name ') ) 

2 db.define_table( 'doctor' , db. person, Field(' specialization' )) 
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It is also possible to define a dummy table that is not stored in a database in 
order to reuse it in multiple other places. For example: 


1 

signature = db.Table(db, 'signature' , 

2 

Field ( 'created-on ' , 'datetime', default=request .now) , 

3 

Field ( 'created-by' , db.auth_user, default=auth.user_id) , 

4 

Field ( ' updated^on ' , 'datetime', update=request.now) , 

5 

Field ( 'updated-by' , db.auth_user, update=auth.user_id) ) 

7 

db.define_table( 'payment' , Field ( 'amount' , 'double'), signature) 


This example assumes that standard web2py authentication is enabled. 

Notice that if you user Auth web2py already creates one such table for you: 

, auth = Auth(db) 

2 db.define_table( 'payment' , Field ( 'amount' , 'double'), auth. signature) 

When using table inheritance, if you want the inheriting table to inherit 
validators, be sure to define the validators of the parent table before defining 
the inheriting table. 


6.29.2 Common fields and multi-tenancy 

db._common_fields is a list of fields that should belong to all the tables. This 
list can also contain tables and it it is understood as all fields from the table. 
For example occasionally you find yourself in need to add a signature to all 
your tables but the 'auth tables. In this case, after you db.define_tables() but 
before defining any other table, insert 

db. _common_ fields. append (auth .signature) 

One field is special: "request_tenant". This field does not exist but you can 
create it and add it to any of your tables (or them all): 

db._ common- fields. append ( Field ( ' request-tenant' , 
default=request .env.http_host,writable=False) ) 

For every table with a field called db._request_tenant, all records for all 
queries are always automatically filtered by: 

db. table. request-tenant == db. table. request-tenant .default 
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and for every record insert, this field is set to the default value. In the 
example above we have chosen 
default = request .env.http_host 

i.e. we have chose to ask our app to filter all tables in all queries with 
db. table. request_tenant == request .env.http_host 

This simple trick allow us to turn any application into a multi-tenant 
application, i.e. even if we run one instance of the app and we use one single 
database, if the app is accessed under two or more domains (in the example 
the domain name is retrieved from request .env.http_host) the visitors will 
see different data depending on the domain. Think of running multiple web 
stores under different domains with one app and one database. 

You can turn off multi tenancy filters using: 

rows = db(query : ignore_common_filters=True) . select( ) 


6.29.3 Common filters 

A common filter is a generalization of the above multi-tenancy idea. It 
provides an easy way to prevent repeating of the same query. Consider for 
example the following table: 

1 db. def ine_table ( 'blog-post' , 
z Field ( 'subject' ) , 

3 Field( 'post-text' , 'text'), 

4 Field ( 'is-public' , 'boolean'), 

5 common_filter = lambda query: db.blog_post .is_public==True 


Any select, delete or update in this table, will include only public blog posts. 
The attribute can also be changed in controllers: 

db.blog_post ._common_filter = lambda query: db.blog_post .is_public == True 

It serves both as a way to avoid repeating the "db.blog_post.is_public==True" 
phrase in each blog post search, and also as a security enhancement, that 
prevents you from forgetting to disallow viewing of none public posts. 
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In case you actually do want items left out by the common filter (for example, 
allowing the admin to see none public posts), you can either remove the filter: 

db.blog_post ._common_filter = None 

or ignore it: 

db(query, ignore_common_filters=True) .select( . . . ) 


6.29.4. Custom Field types (experimental) 

It is possible to define new /custom field types. For example we consider 
here the example if a field that contains binary data in compressed form: 

1 from gluon.dal import SQLCustomType 

2 import zlib 

3 

4 compressed = SQLCustomType( 

5 type =' text' , 

6 native^' text' , 

7 encoder =(lambda x: zlib. compress(x or '')), 

8 decoder = (lambda x: zlib.decompress(x) ) 

9 ) 
10 

u db.define_table( 'example' , Field) 'data ' ,type=compressed) ) 

SQLCustomType is a field type factory. Its type argument must be one of the 
standard web2py types. It tells web2py how to treat the field values at 
the web2py level, native is the name of the field as far as the database is 
concerned. Allowed names depend on the database engine, encoder is an 
optional transformation function applied when the data is stored and decoder 
is the optional reversed transformation function. 

This feature is marked as experimental. In practice is has been in web2py for 
a long time and it works but it can make the code not portable, for example 
when the native type is database specific. It does not work on Google App 
Engine NoSQL. 

6.29.5 Using DAL without define tables 

The DAL can be used from any Python program simply by doing this: 
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from gluon import DAL, Field 

db = DAL( ' sqlite: / /storage, sqlite' , folder=' pat h/to/app/ databases ' ) 

i.e. import the DAL, Field, connect and specify the folder which contains 
the. table files (the app/databases folder). 

To access the data and its attributes we still have to define all the tables we 
are going to access with db . def ine_tables (...). 

If we just need access to the data but not to the web2py table attributes, we 

get away without re-defining the tables but simply asking web2py to read 

the necessary info from the metadata in the. table files: 

from gluon import DAL, Field 

db = DAL( ' sqlite: / /storage. sqlite' , folder=' pat h/to/app/ databases ' , 
auto_import=True) ) 

This allows us to access any db. table without need to re-define it. 

6.29.6 Copy data from one db into another 

Consider the situation in which you have been using the following database: 

db = DAL( 'sqlite : //storage, sqlite' ) 

and you wish to move to another database using a different connection string: 
db = DAL( ' postgresql://username:password@hocalhost/mydb' ) 

Before you switch, you want to move the data and rebuild all the metadata 
for the new database. We assume the new database to exist but we also 
assume it is empty. 

Web2py provides a script that does this work for you: 


1 cd web2py 

2 python scripts/cpdb.py \ 

3 -f applications/app/databases \ 

4 -y ' sqlite: //storage. sqlite' \ 

5 -Y ' postgresql:/ /username:password@hocalhost/mydb' 

After running the script you can simply switch the connection string in the 
model and everything should work out of the box. The new data should be 
there. 
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This script provides various command line options that allows you to move 
data from one application to another, move all tables or only some tables, 
clear the data in the tables, for more info try: 

python scripts/cpdb.py -h 


6.29.J Note on new DAL and adapters 

The source code of the Database Abstraction Layer was completely rewritten 
in 2010. While it stays backward compatible, the rewrite made it more 
modular and easier to extend. Here we explain the main logic. 

The file "gluon/dal.py" defines, among other, the following classes. 

1 ConnectionPool 

z BaseAdapter extends ConnectionPool 

3 Row 

4 DAL 

^ Reference 

6 Table 

7 Expression 

8 Field 

9 Query 
10 Set 

i] Rows 

Their use has been explained in the previous sections, except for BaseAdapter. 
When the methods of a Table or Set object need to communicate with the 
database they delegate to methods of the adapter the task to generate the 
SQL and or the function call. 

For example: 

1 db. myable. insert (myfield='myi/alue' ) 

calls 

] Table. insert (myfield='myi/alue' ) 

which delegates the adapter by returning: 
1 db. .adapter. insert (db.myt able, db.mytable. _list if y( diet (my field= ' myvalue' ) ) ) 
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Here db.mytable._listify converts the diet of arguments into a list of 
(field, value) and calls the insert method of the adapter. db._adapter does 
more or less the following: 

1 query = db. .adapter. -insert (db.mytable,list_of_fields) 

2 db. .adapter. execute(query) 

where the first line builds the query and the second executes it. 

BaseAdapter define the interface for all adapters. 

"gluon/dal.py" at the moment of writing this book, contains the following 

adapters: 

] SQLiteAdapter extends BaseAdapter 

z JDBCSQLiteAdapter extends SQLiteAdapter 

3 MySQLAdapter extends BaseAdapter 

4 PostgreSQLAdapter extends BaseAdapter 

5 JDBCPostgreSQLAdapter extends PostgreSQLAdapter 

6 OracleAdapter extends BaseAdapter 
- MSSQLAdapter extends BaseAdapter 

s MSSQL2Adapter extends MSSQLAdapter 

i, FireBirdAdapter extends BaseAdapter 

[o FireBirdEmbeddedAdapter extends FireBirdAdapter 

u InformixAdapter extends BaseAdapter 

12 DB2Adapter extends BaseAdapter 

[ 3 IngresAdapter extends BaseAdapter 

14 IngresUnicodeAdapter extends IngresAdapter 

[ 5 GoogleSQLAdapter extends MySQLAdapter 

ib NoSQLAdapter extends BaseAdapter 

17 GoogleDatastoreAdapter extends NoSQLAdapter 

18 CubridAdapter extends MySQLAdapter (experimental) 
11, TeradataAdapter extends DB2Adapter (experimental) 

20 SAPDBAdapter extends BaseAdapter (experimental) 

21 CouchDBAdapter extends NoSQLAdapter (experimental) 

22 MongoDBAdapter extends NoSQLAdapter (experimental) 

which override the behavior of the BaseAdapter. 

Each adapter has more or less this structure: 

, class MySQLAdapter(BaseAdapter) : 
2 

3 # specify a diver to use 

4 driver = globals( ) .get ( ' pymysql ', None) 


5 


# map web2py types into database types 
types = { 
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■boolean': 'CHAR(l)', 

■string': ' VARCHAR(%(length)s) ' , 

'text': 'LONGTEXT', 

} 

# connect to the database using driver 

def init (self ,db,uri,pool_ size=0,folder=None,db_codec ='UTF-8' , 

credential_decoder=lambda x:x : driver_args={} : 

adapter_args={}) : 

# parse uri string and store parameters in driver_args 

# define a connection function 

def connect (driver_args=driver_args) : 

return self .drive r. connect (**d rive r_args) 

# place it in the pool 

self .pool_connect ion (connect) 

# set optional parameters (after connection) 
self .execute( 'SET F0REIGN-KEY-CHECKS=1; ' ) 

self . execute ( "SET sql_mode= ' NO^BACKSLASH^ESCAPES ' ; " ) 

# override BaseAdapter methods as needed 
def last rowid( self , table) : 

self .execute) 'select last-insert-id( ) ; ' ) 

return int(self .cursor. fetchone( ) [0] ) 

Looking at the various adapters as examples should be easy to write new 
ones. 

When db instance is created: 

db = DAL( 'mysql://. . . ' ) 

the prefix in the uri string defines the adapter. The mapping is defined in the 
following dictionary also in "gluon/dal.py": 


ADAPTERS = { 

'sqlite': SQLiteAdapter, 

1 sqlite: memory ' : SQLiteAdapter, 

'mysql': MySQLAdapter, 

'postgres': PostgreSQLAdapter, 

'oracle': OracleAdapter, 

'mssql': MSSQLAdapter, 

'mssqW: MSSQL2Adapter, 

'db2' : DB2Adapter, 

' teradata' : TeradataAdapter, 

'informix' : InformixAdapter, 

'firebird': FireBirdAdapter, 
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the uri string is then parsed in more detail by the adapter itself. 

For any adapter you can replace the driver with a different one: 

1 from gluon.dal import MySQLAdapter 

2 MySQLAdapter. driver = mysqldb 

and you can specify optional driver arguments and adapter arguments: 
] db =DAL(..., driver_args={}, adapter_args={}) 


6.29.8 Gotchas 

SQLite does not support dropping and altering columns. That means that 
web2py migrations will work up to a point. If you delete a field from a 
table, the column will remain in the database but be invisible to web2py If 
you decide to re-instate the column, web2py will try re-create it and fail. In 
this case you must set fake_migrate=True so that metadata is rebuilt without 
attempting to add the column again. Also, for the same reason, SQLite is 
not aware of any change of column type. If you insert a number in a string 
field, it will be stored as string. If you later change the model and replace the 
type "string" with type "integer", SQLite will continue to keep the number as 
a string and this may cause problem when you try to extract the data. 

MySQL does not support multiple ALTER TABLE within a single 
transaction. This means that any migration process is broken into multiple 
commits. If something happens that causes a failure it is possible to 
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break a migration (the web2py metadata are no-longer in sync with the 
actual table structure in the database). This is unfortunate but it can be 
prevented (migrate one table at the time) or it can be fixed a posteriori 
(revert the web2py model to what corresponds to the table structure in 
database, set fake_migrate=True and after the metadata has been rebuilt, set 
fake_migrate=False and migrate the table again). 

Google SQL has the same problems as MySQL and some more. In particular 
table metadata itself must be stored in the database in a table that is not 
migrated by web2py. This is because Google App Engine has a readonly 
file system. Web2py migrations in Google:SQL combined with the MySQL 
issue described above can result in metadata corruption. Again this can be 
prevented (my migrating the table at once and then setting migrate=False so 
that the metadata table is not accessed any more) or it can fixed a posteriori 
(my accessing the database using the Google dashboard and deleting any 
corrupted entry from the table called web2py_f ilesystem. 

MSSQL does not support the SQL OFFSET keyword. Therefore the database 
cannot do pagination. When doing a limitby=(a,b) web2py will fetch the first 
b rows and discard the first the a. This may result in a considerable overhead 
when compared with other database engines. 

Oracle also does not support pagination. It does not support neither 
the OFFSET nor the LIMIT keywords. Web2py achieves pagination but 
translating a db( . . . ) . select ( limitby=( a, b) ) into a complex three-way nested 
select (as suggested by official Oracle documentation). This works for simple 
select but may break for complex selects involving alised fields and or joins. 

MSSQL has problems with circular references in tables that have ONDELETE 
CASCADE. This is MSSSQL bug and you work around it by setting the 
ondelete attribute for all reference fields to "NO ACTION". You can also 
do it once for all before you define tables: 



MSSQL also has problems with arguments passed to the DISTINCT keyword 
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and therefore while this works, 
db(query) . select (distinct=True) 

this does not 

db( query) .select (distinct=db. my table. my field) 

Google NoSQL (Datastore) does not allow joins, left joins, aggregates, 
expression, OR involving more than one table, the like operator and search 
in "text"" fields. Transactions are limited and not provided automatically by 
web2py (you need to use the Google API run_in_transaction which you can 
look up in the Google App Engine documentation online). Google also limits 
the number of records you can retrieve in each one query (1000 at the time 
of writing). On the Google datastore record IDs are integer but they are not 
sequential. While on SQL the "list:string" type is mapped into a "text" type, 
on the Google Datastore it is mapped into a ListStringProperty. Similarly 
"list:integer" and "list:reference" are mapped into "ListProperty". This makes 
that searches for content inside these fields types are more efficient on Google 
NoSQL than on SQL databases. 


7 

Forms and validators 


There are four distinct ways to build forms in web2py: 

• FORM provides a low-level implementation in terms of HTML helpers. A 
FORM object can be serialized into HTML and is aware of the fields it 
contains. A FORM object knows how to validate submitted form values. 

• SQLFORM provides a high-level API for building create, update and delete 
forms from an existing database table. 

• SQLFORM. factory is an abstraction layer on top of SQLFORM in order to take 
advantage of the form generation features even if there is no database 
present. It generates a form very similar to SQLFORM from the description 
of a table but without the need to create the database table. 

• CRUD methods. These are functionally equivalent to SQLFORM and are 
based on SQLFORM, but provide a more compact notation. 

All these forms are self -aware and, if the input does not pass validation, they 
can modify themselves and add error messages. The forms can be queried 
for the validated variables and for error messages that have been generated 
by validation. 

Arbitrary HTML code can be inserted into or extracted from the form using 
helpers. 
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FORM and SQLFORM are helpers and they can be manipulated in as similar way 
as the DIV. For example you can set a form style: 

, form = SQLFORM (. .) 

2 form [ 'style' ] = ' border: lpx solid black' 


J.l FORM 

Consider as an example a test application with the following "default.py" 
controller: 

def display_form() : 
return dict() 


and the associated "default/display _form.html" view: 

1 {{extend 'layout.hr/nl'}} 

2 <h2>Input form</h2> 

, <form enctype="multipart/ form-data" 

4 action=" {{=URL()}}" method="post"> 

5 Your name: 

6 <input name="name" /> 

7 <input type=" submit" /> 

8 </form> 

i, <h2>Submitted variables</h2> 
10 {{=BEAUTIFY(request.vars)}} 

This is a regular HTML form that asks for the user's name. When you fill 
the form and click the submit button, the form self -submits, and the variable 
request .vars. name and its value is displayed at the bottom. 

You can generate the same form using helpers. This can be done in the view 
or in the action. Since web2py processed the form in the action, it is OK to 
define the form in the action. 

Here is the new controller: 


def display_form( ) : 





form=F0RM( ' Your name: ' 

, INPUT(_name= 

' name ' ) 

, INPUT (_type= 

'submit' ) ) 

return diet (form=form) 






and the associated "default/display _form.html" view: 
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{{extend 'layout.html'}} 

<h2>Input form</h2> 

{{=form}} 

<h2>Submitted variables</h2> 

{{=BEAUTI FY (request. vars)}} 

The code so far is equivalent to the previous code, but the form is generated 
by the statement {{=form}} which serializes the FORM object. 

Now we add one level of complexity by adding form validation and 
processing. 

Change the controller as follows: 



and the associated "default/display _form.html" view: 

{{extend 'layout.html'}} 
<h2>Input form</h2> 
{{=form}} 

<h2>Submitted variables</h2> 
{{=BEAUTI FY ( request. vars)}} 
<h2>Accepted variables</h2> 
{{=BEAUTIFY(form.vars)}} 
<h2>Errors in form</h2> 
{{=BEAUTIFY( form. errors)}} 

Notice that: 

• In the action, we added the requires=IS_NOT_EMPTY( ) validator for the input 
field "name". 

• In the action, we added a call to form.acceptsf . . ) 

• In the view, we are printing form. vars and form. errors as well as the form 
and request .vars. 
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All the work is done by the accepts method of the form object. It filters 
the request. vars according to the declared requirements (expressed by 
validators), accepts stores those variables that pass validation into form. vars. 
If a field value does not meet a requirement, the failing validator returns an 
error and the error is stored in form. errors. Both form. vars and form. errors 
are gluon. storage. Storage objects similar to request .vars. The former contains 
the values that passed validation, for example: 


form. vars . name = "Max" 


The latter contains the errors, for example: 


form. errors . name = "Cannot be empty!" 


The full signature of the accepts method is the following: 


form.accepts(vars, session=None : formname= ' default ' , 
keepvalues=False : onvalidation=None, 
dbio=True, hideerror=False) : 


The meaning of the optional parameters is explained in the next sub-sections. 

The first argument can be request, vars or request . get_vars or 
request .post_vars or simply request. The latter is equivalent to accepting as 
input the request . post_ vars. 

The accepts function returns True if the form is accepted and False otherwise. 
A form is not accepted if it has errors or when it has not been submitted (for 
example, the first time it is shown). 

Here is how this page looks the first time it is displayed: 
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Here is how it looks upon invalid submission: 
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. 

Errors in form 


r 


Here is how it looks upon a valid submission: 
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^ I ►■ + ^3 , http://127.[).D.l:8aoo/te5t/defaLjlt/display_forr & <U? Gooole 



Input form 

Your name; 

Submitted variables 

name Max 

Accepted variables 

name Max 

Errors in form 


y.1.1 The process and validate methods 

A shortcut for 

f rm. accepts) request. post_vars : session, . . . ) 


IS 


form. process ( . . . ) .accepted 


the latter does not need the request and session arguments (although you 
can specify them optionally), it also differs from accepts because it returns 
the form itself. Internally process calls accepts and passes its arguments to it. 
The value returned by accepts is stored in form. accepted. 

The process function takes some extra argument that accepts does not take: 

• message_onsuccess 

• onsuccess:if equal to 'flash' (default) and the form is accepted it will flash 
the above 'message_onsuccess 


message_onfailure 
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onfailure: if equal to 'flash' (default) and the form fails 
validation, it will flash the above 'message_onfailure 

• next the user to redirect after the form is accepted. 

onsuccess and onfailure can be functions like lambda form: 
do_something(form). 


form.validate( . . . ) 

is a shortcut for 

form.process( . . . ,dbio=False) 

.accepted 


7.1.2 Hidden fields 

When the above form object is serialized by {{=form}}, and because of the 
previous call to the accepts method, it now looks like this: 

method="post"> 



Notice the presence of two hidden fields: "_formkey" and "_formname". 
Their presence is triggered by the call to accepts and they play two different 
and important roles: 

• The hidden field called "_formkey" is a one-time token that web2py uses 
to prevent double submission of forms. The value of this key is generated 
when the form is serialized and stored in the session. When the form is 
submitted this value must match, or else accepts returns False without 
errors as if the form was not submitted at all. This is because web2py 
cannot determine whether the form was submitted correctly. 

• The hidden field called "_formname" is generated by web2py as a name for 
the form, but the name can be overridden. This field is necessary to allow 
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pages that contain and process multiple forms. web2py distinguishes the 
different submitted forms by their names. 

• Optional hidden fields specified as F0RM( . . , hidden=dict( ...)). 

The role of these hidden fields and their usage in custom forms and pages 
with multiple forms is discussed in more detail later in the chapter. 

If the form above is submitted with an empty "name" field, the form does not 
pass validation. When the form is serialized again it appears as: 

<form enctype="multipart/ form-data" action="" method="post"> 

your name: 

<input value="" name="name" /> 

<div class="error">cannot be empty !</div> 

<input type=" submit" /> 

<input value=" 783531473471" type=" hidden" name="_ formkey" /> 

<input value=" default" type="hidden" name="-formname" /> 

</form> 

Notice the presence of a DIV of class "error" in the serialized form. web2py 
inserts this error message in the form to notify the visitor about the field that 
did not pass validation. The accepts method, upon submission, determines 
that the form is submitted, checks whether the field "name" is empty and 
whether it is required, and eventually inserts the error message from the 
validator into the form. 

The base "layout.html" view is expected to handle DIVs of class "error". The 
default layout uses jQuery effects to make errors appear and slide down with 
a red background. See Chapter 11 for more details. 


7.2.3 keepvalues 

The optional argument keepvalues tells web2py what to do when a form is 
accepted and there is no redirection, so the same form is displayed again. 
By default the form is cleared. If keepvalues is set to True, the form is pre- 
populated with the previously inserted values. This is useful when you 
have a form that is supposed to be used repeatedly to insert multiple similar 
records. If the dbio argument is set to False, web2py will not perform any DB 
insert/update after accepting form. If hideerror is set to True and the form 
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contains errors, these will not be displayed when the form is rendered (it will 
be up to you to display them from form. errors somehow. The onvalidation 
argument is explained below. 

7.I.4 onvalidation 

The onvalidation argument can be None or can be a function that takes the 
form and returns nothing. Such a function would be called and passed the 
form, immediately after validation (if validation passes) and before anything 
else happens. The purpose of this function is multifold. It can be used, for 
example, to perform additional checks on the form and eventually add errors 
to the form. It can also be used to compute the values of some fields based on 
the values of other fields. It can be used to trigger some action (like sending 
an email) before a record is created/updated. 

Here is an example: 

db . def ine_table ( ' numbers ' , 
Field('a', 'integer'), 
Field ('ft', 'integer'), 
Field('c', 'integer' , readable=False, writable=False) ) 

def my_form_processing(form) : 

c = form. vars.a * form.vars.b 
if c < 0: 

form. errors. b = ' a*b cannot be negative' 
else: 

form.vars.c = c 

def insert_numbers( ) : 

form = SQLFORM(db. numbers) 

if form. process (onvalidation=my_form_ processing) .accepted: 

session. flash = 'record inserted' 

redirect(URLO) 
return diet (form=form) 


7. 1.5 Detect record change 

When filling a form to edit a record there is a small probability that another 
user may concurrently be editing the same record. So when we save the 
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record we want to check for possible conflicts. This can be done: 

1 db . def ine_table ( ' dog ' , Field ( ' name ' ) ) 
2 

3 def edit_dog( ) : 

4 dog = db. dog (request .args(0) ) or redirect(URL( 'error' ) ) 

5 form=SQLFORM(db.dog,dog) 

6 form. process! detect- record- change=True) 

7 if form. record-Changed: 
s # do something 

i) elif form. accepted: 

10 # do something else 

u else: 

12 # do nothing 

13 return diet (form=form) 


7.1.6 Forms and redirection 

The most common way to use forms is via self-submission, so that the 
submitted field variables are processed by the same action that generated 
the form. Once the form is accepted, it is unusual to display the current page 
again (something we are doing here only to keep things simple). It is more 
common to redirect the visitor to a "next" page. 

Here is the new example controller: 


1 

def 

display_form( ) : 

3 


form = FORH( ' Your name: ' , 

INPUT(_name=' name' , requires=IS_NOT_EMPTY( ) ) , 

4 


INPUT (_type=' submit' )) 

5 


if form. process! ) .accepted : 

6 


session. flash = 'form accepted' 

7 


redirect (URL( 'next')) 

8 


elif form. errors: 

9 


response. flash = 'form has errors' 
else: 

response. flash = 'please fill the form' 

" 


return diet (form=form) 

13 

14 

def 

next( ) : 

15 


return dict() 


In order to set a flash on the next page instead of the current page you must 
use session, flash instead of response, flash. web2py moves the former into 
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the latter after redirection. Note that using session. flash requires that you 
do not session ,forget( ). 


7.1.7 Multiple forms per page 

The content of this section applies to both FORM and SQLFORM objects. It is 
possible to have multiple forms per page, but you must allow web2py to 
distinguish them. If these are derived by SQLFORM from different tables, then 
web2py gives them different names automatically; otherwise you need to 
explicitly give them different form names. Here is an example: 

def two_forms( ) : 

forml = FORM(INPUT(_name='name' , requires=IS_NOT_EMPTY( ) ) , 

INPUT(_type=' submit 1 )) 
form2 = FORM ( INPUT (_name=' name' , requires=IS_NOT_EMPTY( ) ) , 

INPUT(_type=' submit' )) 
if forml. process (formname=' form_one' ) .accepted: 

response. flash = 'form one accepted' 
if form2. process (formname=' form_two' ) .accepted: 

response. flash = 'form two accepted' 
return diet (forml=forml, form2=form2) 

and here is the output it produces: 

nnn test 

[ A I ► ] I + |9 Piltp:// 127.0.0. Iiaaoo/test/default/two_form5 C] (Q,- Google 



When the visitor submits an empty formi, only formi displays an error; if 
the visitor submits an empty form2, only form2 displays an error message. 
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j. 1.8 Sharing forms 

The content of this section applies to both FORM and SQLFORM objects. What 
we discuss here is possible but not recommended, since it is always good 
practice to have forms that self-submit. Sometimes, though, you don't have 
a choice, because the action that sends the form and the action that receives 
it belong to different applications. 

It is possible to generate a form that submits to a different action. This is 
done by specifying the URL of the processing action in the attributes of the 
FORM or SQLFORM object. For example: 


1 

form = FORM(INPUT(_name='name' , requires=IS_NOT_EMPTY( ) ) , 

2 

INPUT (_type=' submit' ) , _action=URL( 'page-two' )) 

3 
4 

def page_one( ) : 

5 

return diet (form=form) 

6 
7 

def page_two() : 

8 

if form. process(session=None, formname=None) .accepted: 

9 

response. flash = 'form accepted' 

:o 

else: 

1 

response. flash = 'there was an error in the form' 

.2 

return diet ( ) 


Notice that since both "page_one" and "page_two" use the same form, we 
have defined it only once by placing it outside of all the actions, in order 
not to repeat ourselves. The common portion of code at the beginning of a 
controller gets executed every time before giving control to the called action. 

Since "page_one" does not call process (nor accepts), the form has no name 
and no key, so you must pass session=None and set formname=None in process, 
or the form will not validate when "page_two" receives it. 


J. 2 SQLFORM 

We now move to the next level by providing the application with a model 
file: 

1 db = DAL( ' sqlite:// storage, sqlite' ) 

2 db.define_table( 'person ' , Field( 'name' , requires=IS_NOT_EMPTY( ) ) ) 
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Modify the controller as follows: 


def display_form( ) : 





form = SQLFORM(db. person) 




if form. process! ) .accepted 




response. flash = 

' form 

accepted' 



elif form. errors: 





response. flash = 

' form 

has errors' 



else: 





response. flash = 

'please fill out 

the 

form' 

return diet (form=form 

) 





The view does not need to be changed. 

In the new controller, you do not need to build a FORM, since the SQLFORM 
constructor built one from the table db. person defined in the model. This 
new form, when serialized, appears as: 

1 <form enctype="multipart/ form-data" action="" method="post"> 

2 <table> 

3 <tr id=" person^name row"> 

4 <tdxlabel id=" person-name label" 

5 for="person_name">Your name: </labelx/td> 

6 <tdxinput type="text" class="string" 

7 name="name" value="" id=" person-name" /></td> 
s <tdx/td> 

9 </tr> 

10 <tr id=" submit- record row"> 

<tdx/td> 
12 <tdxinput value="Submit" type=" submit" /></td> 
, 3 <tdx/td> 

14 </tr> 

15 </table> 

10 <input valuer" 9038845529" type="hidden" name="_ formkey" /> 
17 <input value="person" type=" hidden " name="_formrtame" /> 

[8 </form> 

The automatically generated form is more complex than the previous low- 
level form. First of all, it contains a table of rows, and each row has three 
columns. The first column contains the field labels (as determined from the 
db. person), the second column contains the input fields (and eventually error 
messages), and the third column is optional and therefore empty (it can be 
populated with the fields in the SQLFORM constructor). 

All tags in the form have names derived from the table and field name. 
This allows easy customization of the form using CSS and JavaScript. This 
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capability is discussed in more detail in Chapter 11. 

More important is that now the accepts method does a lot more work for you. 
As in the previous case, it performs validation of the input, but additionally, 
if the input passes validation, it also performs a database insert of the new 
record and stores in form, vars.id the unique "id" of the new record. 

A SQLFORM object also deals automatically with "upload" fields by saving 
uploaded files in the "uploads" folder (after having them renamed safely to 
avoid conflicts and prevent directory traversal attacks) and stores their names 
(their new names) into the appropriate field in the database. After the form 
has been processed, the new filename is available in form. vars. fieldname (i.e., 
it replaces the cgi.FieldStorage object in request .vars. fieldname), so you can 
easily reference the new name right after upload. 

A SQLFORM displays "boolean" values with checkboxes, "text" values with 
textareas, values required to be in a definite set or a database with drop- 
boxes, and "upload" fields with links that allow users to download the 
uploaded files. It hides "blob" fields, since they are supposed to be handled 
differently, as discussed later. 

For example, consider the following model: 



In this case, SQLFORMfdb. person) generates the form shown below: 
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test 



| •* | » + Ohttp://l?7.0.0.]:80D0/test/delau]yd 

sp (J J m ^ T Google 

) 







test 

customize ml 

please fill the form 



Index Edit 





Name: 

Max 


Married : @[ 










Profile: 

a 






Image: 

\ CtiGose File J no file selected 



f Submit J 






Copyright © 2010 - Powered by veb2py 

- 


>« - 

/,. 


The SQLFORM constructor allows various customizations, such as displaying 
only a subset of the fields, changing the labels, adding values to the optional 
third column, or creating UPDATE and DELETE forms, as opposed to 
INSERT forms like the current one. SQLFORM is the single biggest time-saver 
object in web2py 

The class SQLFORM is defined in "gluon/sqlhtmlpy". It can be easily extended 
by overriding its xml method, the method that serializes the objects, to change 
its output. 

The signature for the SQLFORM constructor is the following: 

SQLFORM(table, record = None, 

deletable = False, linkto = None, 
upload = None, fields = None, labels = None, 
col3 = {}, submit_button = 'Submit', 
delete_label = 'Check to delete:', 
showid = True, readonly = False, 
comments = True, keepopts = [], 
ignore_rw = False, record_id = None, 
formstyle = ' table3cols' , 
buttons = ['submit' ] , separator = ' : ', 
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**attributes) 

• The optional second argument turns the INSERT form into an UPDATE 
form for the specified record (see next subsection). 

• If deletable is set to True, the UPDATE form displays a "Check to delete" 
checkbox. The value of the label if this field is set via the delete_label 
argument. 

• submit_button sets the value of the submit button. 

• icLlabel sets the label of the record "id" 

• The "id" of the record is not shown if showid is set to False. 

• fields is an optional list of field names that you want to display. If a list is 
provided, only fields in the list are displayed. For example: 

fields = [ 'name' ] 

• labels is a dictionary of field labels. The dictionary key is a field name and 
the corresponding value is what gets displayed as its label. If a label is not 
provided, web2py derives the label from the field name (it capitalizes the 
field name and replaces underscores with spaces). For example: 

labels = {' name' : 'Your Full Name:'} 

• col3 is a dictionary of values for the third column. For example: 

col3 = {'name' :A( 'what is this?', 

_ h ref = ' h t tp : //www. google. com/ sea rch ?q=de fine : name ' ) } 

• linkto and upload are optional URLs to user-defined controllers that allow 
the form to deal with reference fields. This is discussed in more detail 
later in the section. 

• readonly. If set to True, displays the form as readonly 

• comments. If set to False, does not display the C0I3 comments 

• ignore_ rw. Normally, for a create/update form, only fields marked as 
writable=True are shown, and for readonly forms, only fields marked as 
readable=True are shown. Setting ignore- rw=True causes those constraints 
to be ignored, and all fields are displayed. This is mostly used in the 
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appadmin interface to display all fields for each table, overriding what the 
model indicates. 

• formstyle determines the style to be used when serializing the form in 
html. It can be "table3cols" (default), "table2cols" (one row for label and 
comment, and one row for input), "ul" (makes an unordered list of input 
fields), "divs" (represents the form using ess friendly divs, for arbitrary 
customization), formstyle can also be a function that takes (record_id, 
field_label, field_widget, field_comment) as attributes and returns a TR() 
object. 

• is a list of INPUTS or TAG. BUTTONS (though technically could be any 
combination of helpers) that will be added to a DIV where the submit 
button would go. 

• separator sets the string that separates form labels from form input fields. 

• Optional att ributes are arguments starting with underscore that you want 
to pass to the FORM tag that renders the SQLFORM object. Examples are: 

.action = ' . ' 
.method = 'POST 1 


There is a special attribute. When a dictionary is passed as , its items are 
translated into "hidden" INPUT fields (see the example for the FORM helper in 
Chapter 5). 

form = SQLFORM( ,hidden=. . . ) 

causes the hidden fields to be passed with the submission, no more, no less. 
form.accepts( . . . ) is not intended to read the received hidden fields and 
move them into form.vars. The reason is security, hidden fields can be 
tampered with. So you have to do explicitly move hidden fields from the 
request to the form: 

form.vars. a = request .vars .a 

form = SQLF0RM(. . ., hidden=dict (a='b' ) ) 
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7.2.I SQLFORM and insert/update/delete 

SQLFORM creates a new db record when the form is accepted. Assuming 
form=SQLFORM(db.test), then the id of the last-created record will be accessible 
in myform. vars .id. 

If you pass a record as the optional second argument to the SQLFORM 
constructor, the form becomes an UPDATE form for that record. This means 
that when the form is submitted the existing record is updated and no new 
record is inserted. If you set the argument deletable=True, the UPDATE form 
displays a "check to delete" checkbox. If checked, the record is deleted. 

If a form is submitted and the delete checkbox is checked the attribute 
form. deleted is set to True. 

You can modify the controller of the previous example so that when we pass 
an additional integer argument in the URL path, as in: 

/test/default /display. form/2 

and if there is a record with the corresponding id, the SQLFORM generates an 
UPDATE/DELETE form for the record: 



Line 2 finds the record and line 3 makes an UPDATE/DELETE form. Line 4 
does all the corresponding form processing. 

An update form is very similar to a create form except that it is pre-populated 
with the current record and it previews images. By default deletable = True 
which means the update form will display a "delete record" option. 

Edit forms also contain a hidden INPUT field with name="id" which is used to 
identify the record. This id is also stored server-side for additional security 
and, if the visitor tampers with the value of this field, the UPDATE is not 
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performed and weD2py raises a SyntaxError, "user is tampering with form". 

When a Field is marked with writable=False, the field is not shown in create 
forms, and it is shown readonly in update forms. If a field is marked as 
writable=False and readable=False, then the field is not shown at all, not even 
in update forms. 

Forms created with 

form = SQLF0RM( . . . ,ignore_rw=True) 

ignore the readable and writable attributes and always show all fields. Forms 
in appadmin ignore them by default. 

Forms created with 

form = SQLF0RM( table, record_id : readonly=True) 

always show all fields in readonly mode, and they cannot be accepted. 


7.2.2 sqlform in HTML 

There are times when you want to use SQLFORM to benefit from its form 
generation and processing, but you need a level of customization of the form 
in HTML that you cannot achieve with the parameters of the SQLFORM object, 
so you have to design the form using HTML. 

Now, edit the previous controller and add a new action: 

def display_manual_form( ) : 
form = SQLFORM(db. person) 
if form. process(session=None, formname=' test ') .accepted: 

response. flash = 'form accepted' 
elif form. errors: 

response. flash = 'form has errors' 
else: 

response. flash = 'please fill the form' 
# Note: no form instance is passed to the view 
return dict() 

and insert the form in the associated "default /display _manual_form.html" 
view: 
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{{extend 'layout.html'}} 

<form> 

<ul> 

<li>Your name is <input name=" name" /></li> 
</ul> 

<input type=" submit" /> 

<input type="hidden" name="„_formname" valuer" test" /> 
</form> 

Notice that the action does not return the form because it does not need 
to pass it to the view. The view contains a form created manually in HTML. 
The form contains a hidden field "_formname" that must be the same f ormname 
specified as an argument of accepts in the action. web2py uses the form name 
in case there are multiple forms on the same page, to determine which one 
was submitted. If the page contains a single form, you can set f ormname=None 
and omit the hidden field in the view. 

form. accepts will look inside response, vars for data that matches fields in the 
database table db . person. These fields are declared in the HTML in the format 
<input name="field_name_goes_here" /> 

Note that in the example given, the form variables will be passed on the URL 
as arguments. If this is not desired, the POST protocol will have to be specified. 
Note furthermore, that if upload fields are specified, the form will have to be 
set up to allow this. Here, both options are shown: 
<form enctype="multipart/ form-data" method=" post" > 


j. 2. 3 sqlform and uploads 

Fields of type "upload" are special. They are rendered as INPUT fields of 
type="file". Unless otherwise specified, the uploaded file is streamed in 
using a buffer, and stored under the "uploads" folder of the application using 
a new safe name, assigned automatically. The name of this file is then saved 
into the field of type uploads. 

As an example, consider the following model: 

db.define_table( 'person ' , 

Field ( ' name ' , requires=IS_NOT_EMPTY() ) , 
Field (' image ' , 'upload')) 
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You can use the same controller action "display _form" shown above. 

When you insert a new record, the form allows you to browse for a file. 
Choose, for example, a jpg image. The file is uploaded and stored as: 

applications/ test /uploads/person. image. XXXXX.jpg 

"XXXXXX" is a random identifier for the file assigned by web2py 

Notice that, by default, the original filename of an uploaded file is bi6encoded 
and used to build the new name for the file. This name is retrieved by the 
default "download" action and used to set the content disposition header to the 
original filename. 

Only its extension is preserved. This is a security requirement since the 
filename may contain special characters that could allow a visitor to perform 
directory traversal attacks or other malicious operations. 

The new filename is also stored in form . vars . image. 

When editing the record using an UPDATE form, it would be nice to display 
a link to the existing uploaded file, and web2py provides a way to do it. 

If you pass a URL to the SQLFORM constructor via the upload argument, 
web2py uses the action at that URL to download the file. Consider the 
following actions: 


def display_form( ) : 






record = db.person(request 

. a rg s ( ) ) or 

redi 

rect(URL( 

' index 

)) 

form = SQLFORM (db. person, 1 

-ecord, delet 

able 

=True, 



upload=URL( 

'download' ) ) 





if form. process! ) .accepted 






response. flash = 'form 

accepted' 





elif form. errors: 






response. flash = 'form 

has errors ' 





return diet (form=form) 






def download ( ) : 






return response. download (request, db) 






Now, insert a new record at the URL: 

http : //127 . 0.0 . 1 : 8000/test/def ault/display_f orm 

Upload an image, submit the form, and then edit the newly created record 


336 WEB2PY FULL-STACK WEB FRAMEWORK, 4TH EDITION 

by visiting: 

http : //127 . . . 1 : 8000/test/def ault/display_f orm/3 

(here we assume the latest record has id=3). The form will display an image 
preview as shown below: 

lest 


I < I * J I + |#hn p://lZ7.0.Q.hBQ00/tesi/d£tault/display_for m/3 C ] '. Q,- Google 


test 

customize me! 



Index Edit 

Id: 

3 


Name: 

N'ax 

| 

Married: 




Gender: 
Profile: 

Image: 

Male 

H 

/. 





' Ctioose File 1 no file selet 

ed [fiMO delete] 

Check to delete: 



^Subfnlt) 

J 

Copyrigrit © 2010 - Powered by 

^^^^^^^^ 

1 


This form, when serialized, generates the following HTML: 

i <tdxlabel id=" person-image label" for="person_image">Image: </labe"lx/td> 

2 <tdxdivxinput type=" file" id=" person-image" class=" upload" name=" image" 

3 />[<a href ="/ test /default /download/ person. image. 0246683463831. jpg">file</ a>\ 

4 <input type=" checkbox" name=" image delete" />delete]</divx/tdxtdx/tdx/tr> 

5 <tr id=" delete^record row"xtd><label id=" delete- record label" for="delete_record 

6 >Check to delete :</labelx/tdxtdxinput type=" checkbox" id=" delete_record" 

7 class=" delete" name="delete_this_record" /x/td> 


which contains a link to allow downloading of the uploaded file, and a 
checkbox to remove the file from the database record, thus storing NULL 
in the "image" field. 
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Why is this mechanism exposed? Why do you need to write the download 
function? Because you may want to enforce some authorization mechanism 
in the download function. See Chapter 9 for an example. 

Normally uploaded files are stored into "app/uploads" but you can specify 

an alternate location: 

Field ( 'image' , 'upload', uploadfolde r=' . . . ' ) 

In most operating system, accessig the file system can become slow when 
there are many files in the same folder. If you plan to upload more than 1000 
files you can ask web2py to organize the uploads in subfolders: 

Field ( 'image' , 'upload', uploadseparate=True) 


j. 2. 4 Storing the original filename 

web2py automatically stores the original filename inside the new UUID 
filename and retrieves it when the file is downloaded. Upon download, the 
original filename is stored in the content-disposition header of the HTTP 
response. This is all done transparently without the need for programming. 

Occasionally you may want to store the original filename in a database field. 
In this case, you need to modify the model and add a field to store it in: 



then you need to modify the controller to handle it: 
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Notice that the SQLFORM does not display the "image_filename" field. The 
"display_form" action moves the filename of the request .vars. image into the 
form. vars. image-filename, so that it gets processed by accepts and stored 
in the database. The download function, before serving the file, checks in 
the database for the original filename and uses it in the content-disposition 
header. 


J. 2. 5 autodelete 

The SQLFORM, upon deleting a record, does not delete the physical uploaded 
file(s) referenced by the record. The reason is that web2py does not 
know whether the same file is used /linked by other tables or used for 
other purpose. If you know it is safe to delete the actual file when the 
corresponding record is deleted, you can do the following: 


db.define_table( 

' image ' , 

Field ( ' name ' 

, requires=IS_NOT_EMPTY()), 

Field ('file' 

, 'upload' ,autodelete=True) ) 


The autodelete attribute is False by default. When set to True is makes sure 
the file is deleted when the record is deleted. 


y.2.6 Links to referencing records 

Now consider the case of two tables linked by a reference field. For example: 

1 db.define_table( 'person ' , 

Field (' name ' , requires=IS_NOT_EMPTY() ) ) 

3 db.define_table( 'dog' , 

4 Field ( 'owner' , db. person), 

5 Field (' name ' , requires=IS_NOT_EMPTY() ) ) 

6 db. dog. owner. requires = IS_IN_DB(db, db. person. id, '%(name)s' ) 

A person has dogs, and each dog belongs to an owner, which is a person. 
The dog owner is required to reference a valid db. person. id by '%(name)s'. 

Let's use the appadmin interface for this application to add a few persons 
and their dogs. 


FORMS AND VALIDATORS 339 


When editing an existing person, the appadmin UPDATE form shows a link 
to a page that lists the dogs that belong to the person. This behavior can be 
replicated using the linkto argument of the SQLFORM. linkto has to point to the 
URL of a new action that receives a query string from the SQLFORM and lists 
the corresponding records. Here is an example: 



Here is the page: 



There is a link called "dog. owner". The name of this link can be changed via 
the labels argument of the SQLFORM, for example: 

labels = {' dog. owner' -."This person's dogs"} 

If you click on the link you get directed to: 

/test/default/list_ records/dog?query=dog . owner%3D5 


"list_records" is the specified action, with request .args(O) set to the name 
of the referencing table and request .vars. query set to the SQL query string. 
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The query string in the URL contains the value "dog.owner=5" appropriately 
url-encoded (web2py decodes this automatically when the URL is parsed). 

You can easily implement a very general "list_records" action as follows: 



with the associated "default/list_records.html" view: 

1 {{extend ' layout. html ' }} 

2 {{^records}} 

When a set of records is returned by a select and serialized in a view, it 
is first converted into a SQLTABLE object (not the same as a Table) and 
then serialized into an HTML table, where each field corresponds to a table 
column. 


j. 2. j Pre-populating the form 

It is always possible to pre-populate a form using the syntax: 
form.vars .name = ' fieldvalue' 

Statements like the one above must be inserted after the form declaration and 
before the form is accepted, whether or not the field ("name" in the example) 
is explicitly visualized in the form. 

y.2.8 Adding extra form elements to sqlform 

Sometimes you may wish to add an extra element to your form after it has 
been created. For example, you may wish to add a checkbox which confirms 
the user agrees with the terms and conditions of your website: 
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The variable my_extra_element should be adapted to the formstyle. In this 
example, the default formstyle='table3cols' has been assumed. 

After submission, form. vars. agree will contain the status of the checkbox, 
which could then be used in an onvalidation function, for instance. 


7.2.9 sqlform without database IO 

There are times when you want to generate a form from a database table 
using SQLFORM and you want to validate a submitted form accordingly, but 
you do not want any automatic INSERT/UPDATE/DELETE in the database. 
This is the case, for example, when one of the fields needs to be computed 
from the value of other input fields. This is also the case when you need to 
perform additional validation on the inserted data that cannot be achieved 
via standard validators. 

This can be done easily by breaking: 

1 form = SQLFORM(db. person) 

2 if form. process( ) .accepted: 

3 response. flash = 'record inserted' 

into: 

1 form = SQLFORM(db. person) 

2 if form. validate) ) : 

3 ### deal with uploads explicitly 

4 form. vars. id = db. person. insert(**dict (form. vars) ) 

5 response. flash = 'record inserted' 

The same can be done for UPDATE/DELETE forms by breaking: 

1 form = SQLFORM(db. person, record) 

2 if form. process( ) .accepted: 

3 response. flash = 'record updated' 

into: 
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record . update. record (**dict (form. vars) ) 
response. flash = 'record updated' 

In the case of a table including an "upload"-type field ("fieldname"), both 
process(dbio=False) and validate! ) deal with the storage of the uploaded file 
as if process (dbio=True), the default behavior. 

The name assigned by web2py to the uploaded file can be found in: 

form.vars . fieldname 


J. 3 SQLFORM. factory 

There are cases when you want to generate forms as if you had a database 
table but you do not want the database table. You simply want to take 
advantage of the SQLFORM capability to generate a nice looking CSS-friendly 
form and perhaps perform file upload and renaming. 

This can be done via a form_ factory. Here is an example where you generate 
the form, perform validation, upload a file and store everything in the session 



Here is the "default/form_from_factoryhtml" view: 


1 {{extend 'layout.html'}} 

2 {{=form}} 


You need to use an underscore instead of a space for field labels, or explicitly 
pass a dictionary of labels to fornufactory, as you would for a SQLFORM. 
By default SQLFORM. factory generates the form using html "id" attributes 
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generated as if the form was generated from a table called "no_table". To 
change this dummy table name, use the table_name attribute for the factory: 

form = SQLFORM.factory( . . . ,table_name= 'other_c(ummy_name' ) 

Changing the table_name is necessary if you need to place two factory 
generated forms in the same table and want to avoid CSS conflicts. 


7.3.1 One form for multi-pie tables 

It often happens that you have two tables (for example 'client' and 'address' 
which are linked together by a reference and you want to create a single 
form that allows to insert info about one client and its default address. Here 
is how: model: 

1 db.define_table( ' client ' , 
z Field ( ' name ' ) ) 

3 db.define_table( 'address' , 

4 Field ( 'client' , db. client , writ able=False, readable=False) , 

5 Field C street' ), Field ( 'city' )) 

controller: 



Notice the SQLFORM. factory (it makes ONE form using public fields from 
both tables and inherits their validators too). On form accepts this does two 
inserts, some data in one table and some data in the other. 

This only works when the tables don't have field names in common. 
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74 CRUD 

One of the recent additions to web2py is the Create/Read/Update/Delete 
(CRUD) API on top of SQLFORM. CRUD creates an SQLFORM, but it 
simplifies the coding because it incorporates the creation of the form, the 
processing of the form, the notification, and the redirection, all in one single 
function. 

The first thing to notice is that CRUD differs from the other web2py APIs we 
have used so far because it is not already exposed. It must be imported. It 
also must be linked to a specific database. For example: 

1 from gluon. tools import Crud 
z crud = Crud(db) 

The crud object defined above provides the following API: 


• c rud .tables ( ) returns a list of tables defined in the database. 

• crud.create(db.tablename) returns a create form for table tablename. 

• crud. read(db. tablename, id) returns a readonly form for tablename and 
record id. 

• crud. update(db. tablename, id) returns an update form for tablename and 
record id. 

• crud. delete(db. tablename, id ) deletes the record. 

• crud. select (db. tablename, query) returns a list of records selected from the 
table. 

• crud. search (db. tablename) returns a tuple (form, records) where form is a 
search form and records is a list of records based on the submitted search 
form. 

• crud ( ) returns one of the above based on the request . args ( ) . 

For example, the following action: 

def data(): return dict(form=crud( ) ) 
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would expose the following URLs: 


http://. 
http://. 
http://. 
http://. 
http://. 
http://. 
http://. 


./[app]/ [controller] /data/tables 
■/[app] /[controller] /data/create/ [tablename] 
■ /[app] /[controller] /data/read/ [tablename]/ [id] 
./[app] /[controller] /data/update/ [tablename] /[id] 
■/[app] /[controller] /data/delete/ [tablename] /[id] 
./[app] /[controller] /data/select/ [tablename] 
■/[app] /[controller] /data/search/ [tablename] 


However, the following action: 

1 def create_tablename( ) : 

2 return diet (form=crud.create(db. tablename) ) 

would only expose the create method 
] http://. . ./[app]/[controller]/create_tablename 

While the following action: 

1 def update_tablename( ) : 

z return diet (form=crud. update(db. tablename, request .args(0) ) ) 

would only expose the update method 
] http://. . ./[app]/[controller]/update_tablename/[id] 

and so on. 

The behavior of CRUD can be customized in two ways: by setting some 
attributes of the crud object or by passing extra parameters to each of its 
methods. 


J.4..1 Settings 

Here is a complete list of current CRUD attributes, their default values, and 
meaning: 

To enforce authentication on all crud forms: 

crud. settings .auth = auth 


The use is explained in chapter 9. 
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To specify the controller that defines the data function which returns the crud 
object 

crud. settings . controller = 'default' 

To specify the URL to redirect to after a successful "create" record: 
crud. settings . create_next = URL( 'index') 

To specify the URL to redirect to after a successful "update" record: 
crud. settings . update_next = URL( 'index') 

To specify the URL to redirect to after a successful "delete" record: 
crud. settings .delete_next = URL( 'index') 

To specify the URL to be used for linking uploaded files: 

crud. settings .download_url = URL (' down load' ) 

To specify extra functions to be executed after standard validation procedures 

for crud. create forms: 

crud. settings . create_onvalidation = Storagel_ist( ) 

StorageList is the same as a Storage object, they are both defined in the file 
"gluon/storage.py", but it defaults to [] as opposed to None. It allows the 
following syntax: 
crud. settings . create_onvalidat ion. mytablename. append (lambda form: .... ) 

To specify extra functions to be executed after standard validation procedures 

for crud. update forms: 

crud. settings . update_onvalidation = Storagel_ist( ) 

To specify extra functions to be executed after completion of crud. create 
forms: 

crud. settings . create_onaccept = StorageList ( ) 

To specify extra functions to be executed after completion of crud. update 
forms: 

crud. settings . update_onaccept = StorageList ( ) 
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To specify extra functions to be executed after completion of crud. update if 
record is deleted: 

c rud. settings . update_ondelete = StorageList ( ) 

To specify extra functions to be executed after completion of crud .delete: 
crud. settings .delete_onaccept = StorageList ( ) 

To determine whether the "update" forms should have a "delete" button: 

crud. settings . update_deletable = True 

To determine whether the "update" forms should show the id of the edited 
record: 

crud. settings . showid = False 

To determine whether forms should keep the previously inserted values or 
reset to default after successful submission: 


crud. settings . keepvalues = False 


Crud always detects whether a record being edited has been modified by a 
third party in the time between the time when the form is displayed and the 
time when it is submitted. This behavior is equivalent to 


form 

.process(detect_ 

record.^ 

;hange= 

=True) 

and it is set 

in: 




crud 

.settings 

.detect 

.record. 

^change = True 


and it can be changed /disabled by setting the variable to False. 

You can change the form style by 

crud. settings . formstyle = ' table3cols : or ' table2cols' or 'divs' or 'ul' 

You can set the separator in all crud forms: 

crud. settings .label_separator = ';' 

You can add captcha to forms, using the same convention explained for auth, 
with: 
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crud 

.settings 

, create. 

captcha = 

= None 

crud 

.settings 

. update. 

captcha = 

= None 

crud 

.settings 

. captcha 

= None 



7. 4.2 Messages 

Here is a list of customizable messages: 

crud. messages . submit_button = 'Submit' 


sets the text of the "submit" button for both create and update forms. 

crud. messages .delete. label = 'Check to delete:' 

sets the label of the "delete" button in "update" forms. 
crud. messages . record_created = 'Record Created' 

sets the flash message on successful record creation. 

crud. messages . record_updated = 'Record Updated' 

sets the flash message on successful record update, 
crud. messages . record_deleted = 'Record Deleted' 

sets the flash message on successful record deletion, 
crud. messages . update_log = 'Record %(id)s updated' 

sets the log message on successful record update. 

crud. messages . create_log = 'Record %(id)s created' 

sets the log message on successful record creation. 

c rud. messages . read_log = 'Record %(id)s read' 

sets the log message on successful record read access. 

c rud. messages .delete_log = 'Record %(id)s deleted' 

sets the log message on successful record deletion. 
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Notice that crud. messages belongs to the class gluon. storage. Message which 
is similar to gluon. storage. Storage but it automatically translates its values, 
without need for the T operator. 

Log messages are used if and only if CRUD is connected to Auth as discussed 
in Chapter 9. The events are logged in the Auth table "auth_events". 


7.4.3 Methods 

The behavior of CRUD methods can also be customized on a per call basis. 
Here are their signatures: 

1 c rud. tables ( ) 

2 crud.create(table, next, (invalidation, onaccept, log, message) 

3 crud. read(table, record) 

4 crud.update(table, record, next, onvalidation, onaccept, ondelete, log, message, 

deletable) 

5 crud.delete(table, record_id, next, message) 

6 crud. select (table, query, fields, orderby, limitby, headers, **attr) 

7 crud.search(table, query, queries, query_labels, fields, field_labels, zero, 

showall, chkall) 

• table is a DAL table or a tablename the method should act on. 

• record and record- id are the id of the record the method should act on. 

• next is the URL to redirect to after success. If the URL contains the 
substring "[id]" this will be replaced by the id of the record currently 
created/updated. 

• onvalidation has the same function as SQLFORM(..., onvalidation) 

• onaccept is a function to be called after the form submission is accepted 
and acted upon, but before redirection. 

• log is the log message. Log messages in CRUD see variables in the 
form.vars dictionary such as "%(id)s". 

• message is the flash message upon form acceptance. 

• ondelete is called in place of onaccept when a record is deleted via an 
"update" form. 
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• deletable determines whether the "update" form should have a delete 
option. 

• query is the query to be used to select records. 

• fields is a list of fields to be selected. 

• orderby determines the order in which records should be selected (see 
Chapter 6). 

• limitby determines the range of selected records that should be displayed 
(see Chapter 6). 

• headers is a dictionary with the table header names. 

• queries a list like ['equals', 'not equal', 'contains'] containing the 
allowed methods in the search form. 

• query_labels a dictionary like query_labels=dict (equals='Equals ' ) giving 
names to search methods. 

• fields a list of fields to be listed in the search widget. 

• field-labels a dictionary mapping field names into labels. 

• zero defaults to "choose one" is used as default option for the drop-down 
in the search widget. 

• showall set it to True if you want rows returned as per the query in the 
first call (added after 1.98.2). 

• chkall set it to True to turn on all the checkboxes in the search form (added 
after 1.98.2). 

Here is an example of usage in a single controller function: 

1 # assuming db.define-tablef ' person' , Field) ' name' ) ) 

2 def people () : 

3 form = crud .create(db. person, next=URL( 'index' ) , 

4 message=T( "record created")) 

5 persons = crud. select (db. person, fields=[ 'name' ] , 

6 headers={ 'person. name' : 'Name'}) 

7 return diet (form=form, persons=persons) 

Here is another very generic controller function that lets you search, 
create and edit any records from any table where the tablename is passed 
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request. args(o): 

def manage ( ) : 

table=db [ request .args(0) ] 

form = crud .update (table, request .args(l) ) 

table. id. represent = lambda id, row: \ 

A( 'edit: ' ,id,_href=URL(args=(request .args(0) ,id) ) ) 
search, rows = crud.search(table) 
return diet (form=form,search=search, rows=rows) 

Notice the line table. id. represents . . that tells weD2py to change the 
representation of the id field and display a link instead to the page itself and 
passes the id as request.args(i) which turns the create page into an update 
page. 


y.4.4 Record versioning 

Both SQLFORM and CRUD provides a utility to version database records: 

If you have a table (db.mytable) that needs full revision history you can just 
do: 
] form = SQLFORM (db.mytable, myrecord) .process(onsuccess=auth. archive) 

] form = crud. update(db.mytable, myrecord, onaccept=auth. archive) 

auth. archive defines a new table called db.mytable_archive (the name is 
derived from the name of the table to which it refers) and on updating, it 
stores a copy of the record (as it was before the update) in the created archive 
table, including a reference to the current record. 

Because the record is actually updated (only its previous state is archived), 
references are never broken. 

This is all done under the hood. Should you wish to access the archive table 
you should define it in a model: 

1 db.define_table( ' mytable_archive' , 

2 Field ( 'current-record' , db.mytable) , 
, db.mytable) 

Notice the table extends db.mytable (including all its fields), and adds a 
reference to the current- record. 
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auth. archive does not timestamp the stored record unless your original table 

has timestamp fields, for example: 

1 db. def ine_table ( 'mytable' , 

z Field ( ' created_on ' , ' datetime ' , 

3 default=request . now, update=request . now,writable=False) , 

4 Field( 'created_by' , auth. user, 

5 default=auth.user_id,update=auth. user_id,writable=False) , 

There is nothing special about these fields and you may give them any name 
you like. They are filled before the record is archived and are archived with 
each copy of the record. The archive table name and /or reference field name 
can be changed like this: 



j. 5 Custom forms 

If a form is created with SQLFORM, SQLFORM.factory or CRUD, there are 

multiple ways it can be embedded in a view allowing multiple degrees of 

customization. Consider for example the following model: 

db . def ine_table ( ' image ' , 
Field ( ' name ' ) , 
Field( ' file' , 'upload')) 

and upload action 

def upload_image( ) : 

return diet (form=crud. create(db. image) ) 

The simplest way to embed the form in the view for upload-image is 
{{=form}} 

This results in a standard table layout. If you wish to use a different layout, 
you can break the form into components 
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1 {{=form. custom. begin}} 

2 Image name: <div>{{=form. custom. widget .name}}</div> 
1 Image file: <div>{{=form. custom. widget .file}}</div> 

4 Click here to upload: {{=form. custom. submit}} 

5 {{=form. custom. end}} 


where form, custom. widget [fieldname] gets serialized into the proper widget 
for the field. If the form is submitted and it contains errors, they are 
appended below the widgets, as usual. 

The above sample form is show in the image below. 

Image name: 


Image file: 

I Choose File \ no File selected 


Click here to upload (submit ) 
Notice that a similar result could have been obtained with: 
crud. settings . formstyle=' table2cols ' 


without using a custom form. Other possible formstyles are "table3cols" (the 
default), "divs" and "ul". 

If you do not wish to use the widgets serialized by web2py, you can replace 
them with HTML. There are some variables that will be useful for this: 

• form. custom. label[f ieldname] contains the label for the field. 

• form, custom, comment [fieldname] contains the comment for the field. 

• form, custom. dspval[f ieldname] form-type and field-type dependent 
display representation of the field. 

• form. custom. inpval[f ieldname] form-type and field-type dependent values 
to be used in field code. 

It is important to follow the conventions described below. 
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j. 5.1 CSS conventions 

Tags in forms generated by SQLFORM, SQLFORM.factory and CRUD follow 
a strict CSS naming convention that can be used to further customize the 
forms. 

Given a table "mytable", and a field "myfield" of type "string", it is rendered 

by default by a 

SQLFORM .widgets . st ring .widget 

that looks like this: 

<input type="text" name= "myfield" id="mytaJble„my field" 
class=" string" /> 

Notice that: 

• the class of the INPUT tag is the same as the type of the field. This is 
very important for the jQuery code in "web2py_ajax.html" to work. It 
makes sure that you can only have numbers in "integer" and "double" 
fields, and that "time", "date" and "datetime" fields display the popup 
calendar / datepicker. 

• the id is the name of the class plus the name of the field, joined by one 
underscore. This allows you to uniquely refer to the field via, for example, 
jQueryf '#mytable_myf ield ' ) and manipulate the stylesheet of the field or 
bind actions associated to the field events (focus, blur, keyup, etc.). 

• the name is, as you would expect, the field name. 

j. 5. 2 Hide errors 

Occasionally, you may want to disable the automatic error placement and 
display form error messages in some place other than the default. That can 
be done easily. 

• In the case of FORM or SQLFORM, pass hideerror=True to the accepts 
method. 

• In the case of CRUD, set crud. sett ings.hideerror=T rue 
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You may also want to modify the views to display the error (since they are 
no longer displayed automatically). 

Here is an example where the errors are displayed above the form and not in 
the form. 

1 {{if form. errors: }} 

2 Your submitted form contains the following errors: 

3 <ul> 

4 {{for fieldname in form. errors: }} 

^ <li>{{=fieldname}} error: {{=form.errors[fieldname]}}</li> 

{{pass}} 
- </ul> 
s {{form. errors .clear( )}} 

9 {{pass}} 

10 {{=form}} 


The errors will displayed as in the image shown below. 

Your submitted form contains tho following errors; 
• name error: enter a value 


Name: 


1 




File: 

f Choose File 1 no file selected 






f Submit ) 



This mechanism also works for custom forms. 


j. 6 Validators 

Validators are classes used to validate input fields (including forms generated 
from database tables). 

Here is an example of using a validator with a FORM: 

INPUT(_name='a' , requires=IS_INT_IN_RANGE(0 : 10)) 


Here is an example of how to require a validator for a table field: 

i db.define_table( 'person ' , Field (' name ') ) 
• db. person. name. requires = IS_NOT_EMPTY( ) 
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Validators are always assigned using the requires attribute of a field. A field 
can have a single validator or multiple validators. Multiple validators are 
made part of a list: 

db. person. name. requires = [IS_NOT_EMPTY( ) , 

IS_NOT_IN_DB(db, ' person. name' )] 

Normally validators are called automatically by the function accepts and 
process of a FORM or other HTML helper object that contains a form. They 
are called in the order in which they are listed. 

One can also call validators explicitly for a field: 

db. person. name. validate (value) 

which returns a tuple (value, error) and error is None if no the value validates. 
Built-in validators have constructors that take an optional argument: 
IS_NOT_EMPTY(error_message='cannot be empty') 

error_message allows you to override the default error message for any 
validator. 

Here is an example of a validator on a database table: 

db. person. name. requires = IS_NOT_EMPTY(error_message=' fill this!') 

where we have used the translation operator T to allow for 
internationalization. Notice that default error messages are not translated. 

Mind that the only validators that can be used with list : type fields are: 

• IS_IN_DB(. . . ,multiple=True) 

• IS_IN_SET(. . . ,multiple=True) 

• IS_N0T_ EMPTY () 

• IS_LIST_0F(...) 

The latter can be used to apply any validator to the individual items in the 
list. 
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j. 6.1 Validators 

IS_ALPHANUMERIC 

This validator checks that a field value contains only characters in the ranges 
a-z, A-Z, or 0-9. 

requires = IS_ALPHANUMERIC(error_message='must be alphanumeric! ' ) 

IS_DATE 

This validator checks that a field value contains a valid date in the specified 
format. It is good practice to specify the format using the translation 
operator, in order to support different formats in different locales. 

requires = IS_DATE(format=T( '%y-%m-%d' ) , 

error_message=' must be YYYY-MM-DD! ' ) 

For the full description on % directives look under the IS_DATETIME 
validator. 

IS_DATE_IN_RANGE 

Works very much like the previous validator but allows to specify a range: 

requires = IS_DATE_IN_RANGE( forma t=T( •HY-hm-Hd' ) , 

minimum=datetime. date (2008, 1, 1) , 
maximum=datetime. date (2009, 12,31) , 
error_message=' must be YYYY-MM-DD! ' ) 

For the full description on % directives look under the IS_DATETIME 
validator. 

IS_DATETIME 

This validator checks that a field value contains a valid datetime in the 
specified format. It is good practice to specify the format using the translation 
operator, in order to support different formats in different locales. 

requires = IS_DATETIME(format=T( '%r-%m-%d W:°-M:%S , ) l 

error_message='must be YYYY-MM-DD HH-.MM-.SS! ' ) 

The following symbols can be used for the format string (this shows the 
symbol and an example string): 
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1 

%Y 

'1963' 

2 

%y 

'63' 

3 

%d 

'28' 

4 

%m 

'08' 

5 

%b 

'/Aug' 

6 

%b 

'August' 

7 

%H 

'14' 

8 

%I 

'82' 

9 

%p 

'PM' 


%M 

'30' 

[i 

%S 

'59' 


IS_DATETIME_IN_ RANGE 

Works very much like the previous validator but allows to specify a range: 

1 requires = IS_DATETIME_IN_RANGE(format=T( '%Y-%m-%d %H:*M:%S'), 

minimum=datetime.datetime(2008,l,l,10 : 30) , 

3 maximum=datetime.datetime(2009,12,31 : ll : 45) , 

4 error_message= 'must be YYYY-MM-DD HH:MM: :SS! ' ) 

For the full description on % directives look under the IS_DATETIME 
validator. 

IS_DECIMAL_IN_ RANGE 
, INPUT(_type=' text' , _name='name' , requires=IS_DECIMAL_IN_RANGE(0, 10, dot=".")) 

It converts the input into a Python Decimal or generates an error if the 
decimal does not fall within the specified inclusive range. The comparison is 
made with Python Decimal arithmetic. 

The minimum and maximum limits can be None, meaning no lower or upper 
limit, respectively. 

The dot argument is optional and allows you to internationalize the symbol 
used to separate the decimals. 

I S_ EMAIL 

It checks that the field value looks like an email address. It does not try to 
send email to confirm. 

] requires = IS_EMAIL(error_message='ini/alid email!') 

IS_EQUAL_TO 
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Checks whether the validated value is equal to a given value (which can be 
a variable): 

requires = IS_EQUAL_TO ( request .vars. password, 

error_message='passworc(s do not match') 

IS_EXPR 

Its first argument is a string containing a logical expression in terms of a 
variable value. It validates a field value if the expression evaluates to True. 
For example: 

requires = IS_EXPR( ' int(value)%3==6' , 

error_message=' not divisible by 3') 

One should first check that the value is an integer so that an exception will 
not occur. 

requires = [IS_INT_IN_RANGE(0, 100), IS_EXPR( ' i/alue%3==0' ) ] 


IS_FLOAT_IN_RANGE 


Checks that the field value is a floating point number within a definite range, 
<= value <= 100 in the following example: 


requires = IS_FLOAT_IN_RANGE(0, 100, dot=".", 

error_message=' too small or too large!') 


The dot argument is optional and allows you to internationalize the symbol 
used to separate the decimals. 

IS_INT_IN_ RANGE 

Checks that the field value is an integer number within a definite range, <= 
value < 100 in the following example: 

requires = IS_INT_IN_RANGE(0, 100, 

error_message=' too small or too large!') 

IS_IN_SET 

Checks that the field values are in a set: 

requires = IS_IN_SET( [ 'a ' , 'b' , ' c' ] ,zero=T( 'choose one'), 
error_message= 'must be a or b or c') 
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The zero argument is optional and it determines the text of the option 
selected by default, an option which is not accepted by the IS_IN_SET validator 
itself. If you do not want a "choose one" option, set zero=None. 

The zero option was introduced in revision (1.67.1). It did not break backward 
compatibility in the sense that it did not break applications but it did change 
their behavior since, before, there was no zero option. 

The elements of the set must always be strings unless this validator 
is preceded by IS_INT_II\LRANGE (which converts the value to int) or 
IS_FLOAT_II\LRANGE (which converts the value to float). For example: 

requires = [IS_INT_IN_RANGE(0 : 8), IS_IN_SET( [2, 3, 5, 7], 
error_message='must be prime and less than IB')] 

You may also use a dictionary or a list of tuples to make the drop down list 
more descriptive: 


### Dictionary example: 




requires = IS_IN_SET({ 'A' : 'Apple' , 

'B' 

: 'Banana' , ' C : 

■ 'Cherry '}, zero=None) 

### List of tuples example: 




requires = IS_IN_SET( [ ( 'A ' , 'Apple' 

),( 

' B' , 'Banana' ) , 

, CC, 'Cherry' )]) 


IS_IN SET and Tagging 

The IS_II\LSET validator has an optional attribute multiple=False. If set to 
True, multiple values can be stored in one field. The field should be of type 
list : integer or list : string, multiple references are handled automatically in 
create and update forms, but they are transparent to the DAL. We strongly 
suggest using the jQuery multiselect plugin to render multiple fields. 

Note that when multiple=True, IS_IN_SET will accept zero or more values, 
i.e. it will accept the field when nothing has been selected, multiple can also 
be a tuple of the form (a,b) where a and b are the minimum and (exclusive) 
maximum number of items that can be selected respectively. 

IS_LENGTH 

Checks if length of field's value fits between given boundaries. Works for 
both text and file inputs. 

Its arguments are: 
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• maxsize: the maximum allowed length / size (has default = 255) 

• minsize: the minimum allowed length / size 
Examples: Check if text string is shorter than 33 characters: 

INPUT(_type=' text' , _name='name' , requires=IS_LENGTH(32) ) 

Check if password string is longer than 5 characters: 

INPUT („type=' password' , _name=' name' , requires=IS_LENGTH(minsize=6) ) 

Check if uploaded file has size between 1KB and 1MB: 

INPUT(_type=' file' , _name='name l , requires=IS_LENGTH( 1048576, 1024)) 

For all field types except for files, it checks the length of the value. In the 
case of files, the value is a cookie. FieldStorage, so it validates the length of 
the data in the file, which is the behavior one might intuitively expect. 

IS_LIST_0F 

This is not properly a validator. Its intended use is to allow validations 
of fields that return multiple values. It is used in those rare cases when a 
form contains multiple fields with the same name or a multiple selection box. 
Its only argument is another validator, and all it does is to apply the other 
validator to each element of the list. For example, the following expression 
checks that every item in a list is an integer in the range 0-10: 

requires = IS_LIST_OF(IS_INT_IN_RANGE(0, 10)) 

It never returns an error and does not contain an error message. The inner 
validator controls the error generation. 

I S_ LOWER 

This validator never returns an error. It just converts the value to lower case. 

requires = IS_LOWER() 

IS_MATCH 

This validator matches the value against a regular expression and returns an 
error if it does not match. Here is an example of usage to validate a US zip 
code: 
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requires = IS_MATCH( "-\d{5}(- \d{4}) ?$ ' , 
error_message='not a zip code') 

Here is an example of usage to validate an IPv4 address (note: the IS_IPV4 
validator is more appropriate for this purpose): 

requires = IS_HATCH( '"\d{l,3}(\ . \d{l,3}){3}$' , 
error_message='not an IP address') 

Here is an example of usage to validate a US phone number: 

requires = IS_MATCH( '"!?((- ) \d{3}- ?\ \ (\d{3}\)) \d{3}- ?\d{4}$' , 
error_message='not a phone number') 

For more information on Python regular expressions, refer to the official 
Python documentation. 

IS_MATCH takes an optional argument strict which defaults to False. When 
set to True it only matches the beginning of the string: 



IS_MATCH takes an other optional argument search which defaults to False. 
When set to True, it uses regex method search instead of method match to 
validate the string. 

IS_N0T_ EMPTY 

This validator checks that the content of the field value is not an empty string. 
requires = IS_NOT_EMPTY(error_message=' cannot be empty!') 

IS_TIME 

This validator checks that a field value contains a valid time in the specified 
format. 

requires = IS_TIME(error„message='roust be HH:MM:SS! ' ) 

IS_URL 

Rejects a URL string if any of the following is true: 
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• The string is empty or None 

• The string uses characters that are not allowed in a URL 

• The string breaks any of the HTTP syntactic rules 

• The URL scheme specified (if one is specified) is not 'http' or 'https' 

• The top-level domain (if a host name is specified) does not exist 

(These rules are based on RFC 2616 [68] ) 

This function only checks the URL's syntax. It does not check that the URL 
points to a real document, for example, or that it otherwise makes semantic 
sense. This function does automatically prepend 'http://' in front of a URL 
in the case of an abbreviated URL (e.g. 'google. ca'). 

If the parameter mode='generic' is used, then this function's behavior 
changes. It then rejects a URL string if any of the following is true: 

• The string is empty or None 

• The string uses characters that are not allowed in a URL 

• The URL scheme specified (if one is specified) is not valid 

(These rules are based on RFC 2396 [69] ) 

The list of allowed schemes is customizable with the allowed_schemes 
parameter. If you exclude None from the list, then abbreviated URLs (lacking 
a scheme such as 'http') will be rejected. 

The default prepended scheme is customizable with the prepend_scheme 
parameter. If you set prepend_scheme to None, then prepending will be 
disabled. URLs that require prepending to parse will still be accepted, but 
the return value will not be modified. 

IS_URL is compatible with the Internationalized Domain Name (IDN) 
standard specified in RFC 3490 [70] ). As a result, URLs can be regular strings 
or Unicode strings. If the URL's domain component (e.g. google. ca) contains 
non-US-ASCII letters, then the domain will be converted into Punycode 
(defined in RFC 3492 [71] ). IS_URL goes a bit beyond the standards, 
and allows non-US-ASCII characters to be present in the path and query 
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components of the URL as well. These non-US-ASCII characters will be 
encoded. For example, space will be encoded as'%20'. The Unicode character 
with hex code ox4e86 will become '%4e%86'. 


Examples: 



IS_SLUG 

requires = IS_SLUG(maxlen=80 : check=False, error_message='must be slug') 

If check is set to True it check whether the validated value is a slug (allowing 
only alphanumeric characters and non-repeated dashes). 

If check is set to False (default) it converts the input value to a slug. 

IS_STR0NG 

Enforces complexity requirements on a field (usually a password field) 

Example: 

requires = IS_STRONG(min=10, special=2, upper=2) 

where 

• min is minimum length of the value 

• special is the minimum number of required special characters special 
characters are any of the following ! @#$%&* (){}[]- + 

• upper is the minimum number of upper case characters 

IS-IMAGE 

This validator checks if a file uploaded through the file input was saved in 
one of the selected image formats and has dimensions (width and height) 
within given limits. 
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It does not check for maximum file size (use IS_LENGTH for that). It returns 
a validation failure if no data was uploaded. It supports the file formats BMP, 
GIF, JPEG, PNG, and it does not require the Python Imaging Library. 

Code parts taken from ref. [72] 

It takes the following arguments: 

• extensions: iter able containing allowed image file extensions in lowercase 

• maxsize: iterable containing maximum width and height of the image 

• minsize: iterable containing minimum width and height of the image 
Use (-1, -1) as minsize to bypass the image-size check. 

Here are some Examples: 

• Check if uploaded file is in any of supported image formats: 

requires = IS_IMAGE() 

• Check if uploaded file is either JPEG or PNG: 

requires = IS_IMAGE(extensions=( 'jpeg' , 'png')) 

• Check if uploaded file is PNG with maximum size of 200x200 pixels: 

requires = IS_IMAGE(extensions=( 'png' ) , maxsize=(200, 200)) 

• Note: on displaying an edit form for a table including requires = 
IS_IMAGE(), a delete checkbox will NOT appear because to delete the file 
would cause the validation to fail. To display the delete checkbox use this 
validation: 

requires = IS_EMPTY_OR(IS_IMAGE( ) ) 

IS_UPLOAD_FILENAME 

This validator checks if the name and extension of a file uploaded through 
the file input matches the given criteria. 

It does not ensure the file type in any way. Returns validation failure if no 
data was uploaded. 

Its arguments are: 
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• filename: filename (before dot) regex. 

• extension: extension (after dot) regex. 

• lastdot: which dot should be used as a filename / extension separator: 
True indicates last dot (e.g., "file.tar.gz" will be broken in "file. tar" + "gz") 
while False means first dot (e.g., "file.tar.gz" will be broken into "file" + 
"tar.gz"). 

• case: o means keep the case; 1 means transform the string into lowercase 
(default); 2 means transform the string into uppercase. 

If there is no dot present, extension checks will be done against an empty 
string and filename checks will be done against the whole value. 

Examples: 

Check if file has a pdf extension (case insensitive): 

requires = IS_UPLOAD_FILENAME(extension='pdf' ) 

Check if file has a tar.gz extension and name starting with backup: 

requires = IS_UPLOAD_FILENAME(filename='Jbac/(up. * ' , extension^' tar.gz' , lastdot= 
False) 

Check if file has no extension and name matching README (case sensitive): 

requires = IS_UPLOAD_FILENAHE ( f ilename= , "README$' , extension '"$ ' , case=0) 
IS_IPV4 

This validator checks if a field's value is an IP version 4 address in decimal 
form. Can be set to force addresses from a certain range. 

IPv4 regex taken from ref. [73] Its arguments are: 

• minip lowest allowed address; accepts: str, e.g., 192.168.0.1; iterable of 
numbers, e.g., [192, 168, o, 1]; int, e.g., 3232235521 

• maxip highest allowed address; same as above 

All three example values are equal, since addresses are converted to integers 
for inclusion check with following function: 

number = 16777216 * IP[0] + 65536 * IP[1] + 256 * IP[2] + IP[3] 
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Examples: 

Check for valid IPv4 address: 
requires = IS_IPU4() 


Check for valid private network IPv4 address: 

requires = IS_IPU4(minip='I92. 168.6.1 ' , maxip='192.I68.255.255' ) 

I S_ LOWER 

This validator never returns an error. It converts the value to lower case. 

requires = IS_LOWER() 

IS-UPPER 

This validator never returns an error. It converts the value to upper case, 
requires = IS_UPPER() 

IS_NULL_0R 

Deprecated, an alias for IS_EMPTY_OR described below. 

I S_ EMPTY_ OR 

Sometimes you need to allow empty values on a field along with other 
requirements. For example a field may be a date but it can also be empty. 
The IS_EMPTY_OR validator allows this: 

requires = IS_EMPTY_OR(IS_DATE( ) ) 

CLEANUP 

This is a filter. It never fails. It just removes all characters whose decimal 
ASCII codes are not in the list [10, 13, 32-127]. 

requires = CLEANUP!) 

CRYPT 

This is also a filter. It performs a secure hash on the input and it is used to 
prevent passwords from being passed in the clear to the database. 

requires = CRYPT ( ) 
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If a key is not specified, it uses the MD5 algorithm. If a key is specified 
CRYPT uses the HMAC algorithm. The key may contain a prefix that 
determines the algorithm to use with HMAC, for example SHA512: 
1 requires = CRYPT( key= ' sha512: thisisthekey' ) 

This is the recommended syntax. The key has to be a unique string associated 
to the database used. The key can never be changed. If you lose the key the 
previously hashed values become useless. 

The CRYPT validator hashed the input and this makes it somewhat special. 
If you need to validate a password field, before it is hash, you can use CRYPT 
in a list of validators but must make sure it is the last of the list so that it is 
called last. For example: 

■ requires = [IS_STR0NG( ) , CRYPT (key='sha5I2: thisisthekey ' ) ] 

CRYPT also takes a min_length argument which defaults to zero. 
7.6.2 Database validators 

IS_NOT_IN_DB 

Consider the following example: 

1 db.define_table( 'person ' , Field (' name ') ) 

2 db. person. name. requires = IS_NOT_IN_DB(db, ' person. name' ) 

It requires that when you insert a new person, his/her name is not already 
in the database, db, in the field person. name. As with all other validators 
this requirement is enforced at the form processing level, not at the database 
level. This means that there is a small probability that, if two visitors try to 
concurrently insert records with the same person. name, this results in a race 
condition and both records are accepted. It is therefore safer to also inform 
the database that this field should have a unique value: 

1 db.define_table( 'person ' , Field (' name ' , unique=True) ) 

2 db. person. name. requires = IS_NOT_IN_DB(db, ' person. name' ) 

Now if a race condition occurs, the database raises an OperationalError and 
one of the two inserts is rejected. 
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The first argument of IS_NOT_IN_DB can be a database connection or a Set. In 
the latter case, you would be checking only the set defined by the Set. 

The following code, for example, does not allow registration of two persons 
with the same name within 10 days of each other: 

import datetime 

now = datetime. datetime. today ( ) 

db.define_table( 'person ' , 

Field ( ' name ' ) , 

Field ( ' registration-Stamp ' , ' datetime ' , def ault=now) ) 
recent = db(db. person. regis t rat ion_stamp>now- datetime. timedelta( 10) ) 
db. person. name. requires = IS_NOT_IN_DB( recent , ' person. name' ) 

IS_IN_DB 

Consider the following tables and requirement: 



It is enforced at the level of dog INSERT/UPDATE/DELETE forms. It 
requires that a dog. owner be a valid id in the field person. id in the database 
db. Because of this validator, the dog . owner field is represented as a dropbox. 
The third argument of the validator is a string that describes the elements in 
the dropbox. In the example you want to see the person %(name)s instead of 
the person %( Id ) s .%(...) s is replaced by the value of the field in brackets for 
each record. 

The zero option works very much like for the IS_IN_SET validator. 

The first argument of the validator can be a database connection or a DAL 
Set, as in IS_NOT_IN_DB. This can be useful for example when wishing to limit 
the records in the drop-down box. In this example, we use IS_II\LDB in a 
controller to limit the records dynamically each time the controller is called: 

def index( ) : 
(...) 
query = (db. table. field == 'xyz') #in practice 'xyz' would be a variable 

db. table. field. requires=IS_IN_DB(db( query) ) 

form=SQLFORM(. . .) 

if form. process! ) .accepted : ■■■ 

(...) 
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If you want the field validated, but you do not want a dropbox, you must put 
the validator in a list. 

db. dog. owner. requires = [IS_IN_DB(db, ' person. id' , '%(name)s' )] 

Occasionally you want the drop-box (so you do not want to use the list 
syntax above) yet you want to use additional validators. For this purpose 
the IS_II\LDB validator takes an extra argument _and that can point to a list of 
other validators applied if the validated value passes the IS_II\LDB validation. 
For example to validate all dog owners in db that are not in a subset: 


subset=db(db. 

person. id>100) 

db. dog. owner. 

requires = IS_IN_DB(db, ' person. id' , '%(name)s' , 


_and=IS_NOT_IN_DB( subset, 'person. id' )) 


IS_IN_DB also takes a cache argument that works like the cache argument of 
select. 

IS_IN„DB and Tagging 

The IS_IN_DB validator has an optional attribute multiple=False. If set to True 
multiple values can be stored in one field. This field should be of type 
list: reference as discussed in Chapter 6. An explicit example of tagging 
is discussed there, multiple references are handled automatically in create 
and update forms, but they are transparent to the DAL. We strongly suggest 
using the jQuery multiselect plugin to render multiple fields. 


j. 6. 3 Custom validators 

All validators follow the prototype below: 

class sample_validator: 

def init (self, *a, error_message='error' ) : 

self. a = a 

self.e = error_message 
def call (self, value): 

if validate(value) : 

return (parsed(value) , None) 

return (value, self.e) 
def formatter(self , value): 

return format(value) 


FORMS AND VALIDATORS J]\ 


i.e., when called to validate a value, a validator returns a tuple (x, y). If y 
is None, then the value passed validation and x contains a parsed value. For 
example, if the validator requires the value to be an integer, x is converted 
to int( value). If the value did not pass validation, then x contains the input 
value and y contains an error message that explains the failed validation. 
This error message is used to report the error in forms that do not validate. 

The validator may also contain a formatter method. It must perform the 

opposite conversion to the one the call does. For example, consider the 

source code for IS_DATE: 

class IS_DATE( object) : 

def init (self, format='%)'-%m-%a" , error_message='must be YYYY-MM-DD! ' ) : 

self. format = format 

self .error_message = error_message 

def call (self, value): 

try: 

y, m, d, hh, mm, ss, t0, tl, t2 = time.strptime(value, str(self. format) 

) 
value = datetime.date(y, m, d) 
return (value, None) 
except: 

return (value, self .error_message) 
def formatter(self , value): 

return value. st rf time (str( self .format) ) 

On success, the call method reads a date string from the form and 

converts it into a datetime.date object using the format string specified in the 
constructor. The formatter object takes a datetime.date object and converts 
it to a string representation using the same format. The formatter is called 
automatically in forms, but you can also call it explicitly to convert objects 
into their proper representation. For example: 



When multiple validators are required (and stored in a list), they are executed 
in order and the output of one is passed as input to the next. The chain breaks 
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when one of the validators fails. 

Conversely, when we call the formatter method of a field, the formatters of 
the associated validators are also chained, but in reverse order. 

Notice that as alternative to custom validators, you can also use the 
onvalidate argument of form. accepts! ... ), form.processf . . . ) and 
form. validate ( . . . ). 


7. 6. 4 Validators with dependencies 

Usually validators are set once for all in models. 

Occasionally, you need to validate a field and the validator depends on the 
value of another field. This can be done in various ways. It can be done in 
the model or in the controller. 

For example, here is a page that generates a registration form that asks for 
username and password twice. None of the fields can be empty, and both 
passwords must match: 



The same mechanism can be applied to FORM and SQLFORM objects. 


y.y Widgets 


Here is a list of available web2py widgets: 

SQLFORM .widgets . st ring .widget 
SQLFORM .widgets . text .widget 
SQLFORM. widgets .pas sword. widget 
SQLFORM. widgets .integer. widget 
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SQLFORM. widgets .double. widget 
SQLFORM. widgets .time. widget 
SQLFORM. widgets .date. widget 
SQLFORM .widgets . da tetime .widget 
SQLFORM. widgets .upload. widget 
SQLFORM .widgets . boolean .widget 
SQLFORM. widgets .opt ions. widget 
SQLFORM. widgets .multiple. widget 
SQLFORM. widgets . radio. widget 
SQLFORM. widgets .checkboxes .widget 
SQLFORM. widgets .autocomplete 

The first ten of them are the defaults for the corresponding field types. The 
"options" widget is used when a field's requires is IS_IN_SET or IS_IN_DB with 
multiple=False (default behavior). The "multiple" widget is used when a 
field's requires is IS_IN_SET or IS_IN_DB with multiple=True. The "radio" and 
"checkboxes" widgets are never used by default, but can be set manually. The 
autocomplete widget is special and discussed in its own section. 

For example, to have a "string" field represented by a textarea: 

Field (' comment ' , 'string', widget=SQLFORM. widgets. text .widget) 

Widgets can also be assigned to fields a posteriori: 
db.mytable.myfield. widget = SQLFORM. widgets . string. widget 

Sometimes widgets take additional arguments and one needs to specify their 

values. In this case one can use lambda 

db.mytable.myfield. widget = lambda field, value: \ 

SQLFORM. widgets. string. widget (field, value, _style=' color -.blue' ) 

Widgets are helper factories and their first two arguments are always field 
and value. The other arguments can include normal helper attributes such as 
-style, -class, etc. Some widgets also take special arguments. In particular 
SQLFORM. widgets, radio and SQLFORM. widgets . checkboxes take a style argument 
(not to be confused with -style) which can be set to "table", "ul", or "divs" in 
order to match the formstyle of the containing form. 

You can create new widgets or extend existing widgets. 

SQLFORM. widgets [type] is a class and SQLFORM. widgets[type] .widget is a static 
member function of the corresponding class. Each widget function takes two 
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arguments: the field object, and the current value of that field. It returns 
a representation of the widget. As an example, the string widget could be 
recoded as follows: 

1 def my_string_widget( field, value): 

2 return INPUT(_name=field. name, 

3 _id="%s_%s" % (field. _tablename, field. name) , 

4 _class=field . type, 

5 _value=value, 

6 requires=field . requires) 
7 

8 Field (' comment ' , 'string', widget=my_string_widget) 

The id and class values must follow the convention described later in this 
chapter. A widget may contain its own validators, but it is good practice to 
associate the validators to the "requires" attribute of the field and have the 
widget get them from there. 


7.7.1 Autocomplete widget 

There are two possible uses for the autocomplete widget: to autocomplete a 
field that takes a value from a list or to autocomplete a reference field (where 
the string to be autocompleted is a representation of the reference which is 
implemented as an id). 

The first case is easy: 

1 db. def ine_t able ( ' category' , Field ( 'name' ) ) 

z db . def ine_table ( ' product ' , Field ( ' name ' ) , Field ( ' category ' ) ) 

3 db. product . category. widget = SQLFORM. widgets .autocomplete( 

4 request, db. category. name, limitby=(0,10) , min_length=2) 

Where limitby instructs the widget to display no more than 10 suggestions 
at the time, and mi n_ length instructs the widget to perform an Ajax callback 
to fetch suggestions only after the user has typed at least 2 characters in the 
search box. 

The second case is more complex: 

1 db. def ine_t able ( ' category' , Field ( 'name' ) ) 

2 db . def ine_table ( ' product ' , Field ( ' name ' ) , Field ( ' category ' ) ) 

3 db. product . category. widget = SQLFORM. widgets .autocomplete( 

4 request, db. category. name, id_field=db. category. id) 
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In this case the value of icLfield tells the widget that even if the value 
to be autocompleted is a db. category. name, the value to be stored is 
the corresponding db. category .id. An optional parameter is orderby that 
instructs the widget on how to sort the suggestions (alphabetical by default). 

This widget works via Ajax. Where is the Ajax callback? Some magic is going 
on in this widget. The callback is a method of the widget object itself. How is 
it exposed? In web2py any piece of code can generate a response by raising 
an HTTP exception. This widget exploits this possibility in the following way: 
the widget sends the Ajax call to the same URL that generated the widget in 
the first place and puts a special token in the request.vars. Should the widget 
get instantiated again, it finds the token and raises an HTTP exception that 
responds to the request. All of this is done under the hood and hidden to the 
developer. 


J. 8 SQLFORM.grid and SQLFORM. smartgrid (experimental) 


These are two high level gadgets that create complex CRUD controls. They 
provide pagination, the ability to browser, search, sort, create, update and 
delete records from a single gadgets. 

The simplest of the two is SQLFORM.grid. Here is an example of usage: 


1 @auth. requires_login() 

2 def manage_users() : 

3 grid = SQLFORM. grid(db.auth_user) 

4 return locals() 


which produces the following page: 
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The first argument of SQLFORM.grid can be a table or a query. The grid gadget 
will provide access to records matching the query. 

Before we dive into the long list of arguments of the grid gadget we need to 

understand how it works. The gadget looks at request . a rgs in order to decide 

what to do (browse, search, create, update, delete, etc.). Each button created 

by the gadget links the same function (manage_users in the above case) but 

passes different request .a rgs. By default all the URL generated by the grid 

are digitally signed and verified. This means one cannot perform certain 

actions (create, update, delete) without being logged-in. These restrictions 

can be relaxed: 

def manage_users( ) : 

grid = SQLFORM.grid(db.auth_user,user_signature=Fal_se) 
return locals () 

but we do not recommend it. 

Because of the way grid works one can only have one grid per controller 
function, unless they are embedded as components via LOAD. 

Because the function that contains the grid may itself manipulate the 
command line arguments, the grid needs to know which args should be 
handled by the grid and which not. For example here is an example of code 
that allows one to manage any table: 

@auth. requires_login( ) 
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the args argument of the grid specifies which request. args should be passed 
along and ignored by the gadget. In our case request . a rgs [ : 1 ] is the name of 
the table we want to manage and it is handled by the manage function itself, 
not by the gadget. 

The complete signature for the grid is the following: 


1 

SQLFORM.grid(query, 

2 

fields=None, 

3 

field_id=None, 

4 

left=None, 

5 

headers={}, 

6 

orderby=None, 

7 

searchable=True, 

8 

sortable=True, 

9 

deletable=True, 

10 

editable=True, 

11 

details=True, 

12 

create=True, 

13 

csv=True, 

14 

paginate=20, 

15 

selectable=None, 

16 

links=None, 

*7 

upload = ' <default>' , 

18 

args=[], 

19 

user_signature = True, 

20 

maxtextlengths={} , 

21 

maxtextlength=20. 

22 

onvalidation=None, 

23 

oncreate=None, 

24 

onupdate=None, 

25 

ondelete=None, 

26 

sorter_icons=( ' ["] ' , ' [v] ' ) , 

27 

ui = l web2py' , 

28 

showbuttontext=True, 

29 

search_widget='default ' , 

30 

_class="web2py_grid" , 

31 

formname=' web2py^grid' , 

32 

ignore_rw = False, 

33 

formstyle = ' table3cols' ) : 
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• fields is a list of fields to be fetched from the database. It is also used to 
determine which fields to be shown in the grid view. 

• fielcLid must be the field of the table to be used as ID, for example 

db.mytable. id. 

• headers is a dictionary that maps 'tablename. fieldname' into the 
corresponding header label. 

• left is an optional left join expressions used to build . . . select (left=. . . ). 

• orderby is used as default ordering for the rows. 

• searchable, sortable, deletable, details, create determine whether one can 
search, sort, delete, view details, and create new records respectively. 

• csv if set to true allows to download the grid in CSV. 

• paginate sets the max number of rows per page. 

• links is used to display new columns which can be links to other pages. 
The links argument must be a list of diet (header=' name' , body=lambda row: 
A(. . .)) where header is the header of the new column and body is a 
function that takes a row and returns a value. In the example, the value is 
a A( . . . ) helper. 

• maxtextlength sets the maximum length of text to be displayed for each 
field value, in the grid view. This value can be overwritten for each field 
using maxtextlengths, a dictionary of 'tablename. fieldname':length. 

• onvalidation, oncreate, onupdate and ondelete are callback functions. All 
but ondelete take a form object as input. 

• sorter_icons is a list of two strings (or helpers) that will be used to 
represent the up and down sorting options for each field. 

• ui can be set equal to 'web2py' and will generate web2py friendly class 
names, can be set equal to j query- ui and will generate jquery UI friendly 
class names, but it can also be its own set of class names for the various 
grid components: 

ui = diet (widget=' ' , 
header^' ' , 
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• search_widget allows to override the default search widget and we refer 
the reader the source code in "gluon/sqlhtml.py" for details. 

• showbutton allows to turn off all buttons. 

• _ class is the class for the grid container. 

• formname, ignore_rwand formstyle are passed to the SQLFORM objects used 
by the grid for create /update forms. 

deletable, editable and details are usually boolean values but they can 
be functions which take the row object and decide whether to display the 
corrsponding button or not. 

A SQLFORM. smartg rid looks a lot like a grid, in fact it contains a grid but it is 
designed to take as input not a query but only one table and to browse said 
table and selected referencing tables. 

For example consider the following table structure: 

1 db . def ine_table ( ' parent ' , Field ( ' name ' ) ) 

2 db.define_table( ' child' , Field ( 'name' ) , Field ( 'parent' , ' reference parent' ) ) 

With SQLFORM. grid you can list all parents: 

, SQLFORM. g rid (db. pa rent) 

all children: 

, SQLFORM. grid(db. child) 
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and all parents and children in one table: 

SQLFORM. g rid ( d b. pa rent, left=db. child. on (db. child. pa rent=db. pa rent. id) ) 

With SQLFORM. smartgrid you can put all the data in one gadget that spawns 

both tables: 

@auth. requires_login( ) : 
def manage ( ) : 

grid = SQLFORM. smartgridfdb. parent, linked_tables=[ 'child' ] ) 

return locals( ) 


which looks like this: 
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Notice the extra "children" links. One could create the extra links using a 
regular grid but they would point to a different action. With a smartgrid they 
are created automatically and handled by the same gadget. 

Also notice that when clicking on the "children" link for a given parent one 

only gets the list of children for that parent (and that is obvious) but also 

notice that if one now tried to add a new child, the parent value for the new 

child is automatically set to the selected parent (displayed in the breadcrumbs 

associated to the gadget). The value of this field can be overwritten. We can 

prevent this by making it readonly: 

1 @auth. requires_login( ) : 

a def manage ( ) : 

1 db. child. parent .writable = False 

4 grid = SQLFORM. smartgridfdb. parent, linked_tables=[ 'child' ] ) 

5 return locals ( ) 
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If the linkecLtables argument is not specified all referencing tables are 
automatically linked. Anyway, to avoid accidentally exposing data we 
recommend explicitly listing tables that should be linked. 

The following code creates a very powerful management interface for all 
tables in the system: 

1 @auth. requires_membership( 'managers ' ) : 

2 def manage () : 

3 table = request .args(0) or ' auth_user' 

4 if not table in db.tables( ) : redirect ( URL ( 'error' ) ) 

5 grid = SQLFORM. smartgrid(db[table] ,args=request .args[ :1] ) 

6 return locals () 

The smartgrid takes the same arguments as a grid and some more with some 
caveats: 

• The first argument is a table, not a query 

• There is a extra argument constraints which is a dictionary of 
'tablename':query which can be used to further restrict access to the 
records displayed in the 'tablename' grid. 

• There is a extra argument linked_tabl.es which is a list of tablenames of 
tables that should be accessible via the smartgrid. 

• All the arguments but the table, args, linked_tables and user_signatures 
can be dictionaries as explained below. 

Consider the previous grid: 
1 grid = SQLFORM. smartgrid(db. parent, linked_tables=[ 'child' ] ) 

It allows one to access both a db. parent and a db. child. Apart for navigation 
controls, for each one table, a smarttable is nothing but a grid. This means 
that, in this case, one smartgrid can create a grid for parent and one grid for 
child. We may want to pass different sets of parameters to these grids. For 
example different sets of searchable parameters. 

While for a grid we would pass a boolean: 
1 grid = SQLFORM. grid(db. parent , searchable=True) 

for a smartgrid we would pass a dictionary of booleans: 
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1 grid = SQLFORM. smartgrid(db. parent, linked_tables=[ ' child' ] , 

2 searchable= dict(parent=True, child=False) ) 

In this way we made parents searchable but children for each parent not 
searchable (there should not be that many to need the search widget). 

The grid and smartgrid gadgets are here to stay but they are marked 
experimental because the actual html layout of what they return and the exact 
set of parameters one can pass to them may be subject to change as new 
functionalities are added. 

grid and smartgrid do not automatically enforce access control like crud does 
but you can integrate it with auth using explicit permission checking: 

1 grid = SQLFORM. grid(db.auth_user, 

z editable = auth.has_membership( 'managers' ) , 

3 deletable = auth.has_membership( 'managers' ) ) 

or 

1 grid = SQLFORM. grid(db.auth_user, 

2 editable = auth.has_permission( 'edit' , ' auth-user' ) , 

3 deletable = auth. has_permission( 'delete' , ' auth-user' ) ) 

The smartgrid is the only gadget in web2py that displays the table name and 
it need both the singular and the plural. For example one parent can have 
one "Child" or many "Children". Therefore a table object needs to know its 
own singular and plural names. web2py normally guesses them but you can 
set the explicitly: 
] db.define_table( ' child' , ..., singular="Chilcf" , plural="Chilc(ren") 

or with: 


1 

db. 

.define. 

.tablef ' 

child', ...) 

2 

db. 

.child.. 

_singulc 

ir = "Child" 

3 

db, 

.child.. 

.plural 

= "Children" 


They should also be internationalized using the T operator. 

The plural and singular values are then used by smartgrid to provide correct 
names for headers and links. 


8 

Email and SMS 


8.1 Setting up email 

Web2py provides the gluon. tools. Mail class to make it easy to send emails 
using web2py. One can define a mailer with 



Note, if your application uses Auth (discussed in the next chapter), the auth 
object will include its own mailer in auth . settings . mailer, so you can use that 
instead as follows: 



You need to replace the mail. settings with the proper parameters for your 
SMTP server. Set mail. settings. login=False if the SMTP server does not 
require authentication. 

For debugging purposes you can set 
i mail. settings. server = 'logging' 
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and emails will not be sent but logged to the console instead. 

8.1.1 Configuring email for Google App Engine 

For sending emails from Google App Engine account: 

] mail. settings . server = 'gae' 

At the time of writing web2py does not support attachments and encrypted 
emails on Google App Engine. 

8.1.2 X509 and PGP Encryption 

It is possible to send X509 (SMIME) encrypted emails using the following 
settings: 

1 mail. settings . cipher_type = 'x509' 

2 mail. settings . sign = True 

3 mail. settings . sign_passphrase = 'your passphrase' 

4 mail. settings .encrypt = True 

5 mail. settings .x509_sign_keyfile = ' filename. key' 

6 mail. settings .x509_sign_certfile = ' filename. cert' 

7 mail. settings .x509_crypt_certfiles = ' filename. cert' 

It is possible to send PGP encrypted emails using the following settings: 

1 from gpgme import pgp 

2 mail. settings. cipher_type = 'gpg' 

3 mail. settings . sign = True 

4 mail. settings . sign_passphrase = 'your passphrase' 

5 mail. settings .encrypt = True 


The latter requires the python-pyme package. 


8.2 Sending emails 


Once mail is defined, it can be used to send email via: 

mail. send (to=[ ' somebodyisexample . com' ] , 
subject='heI!o' , 
# If reply-to is omitted, then mail. settings. sender is used 
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4 reply_to= 'us@example. com' , 

5 message='/ii there') 

Mail returns True if it succeeds in sending the email and False otherwise. A 
complete argument list for mail . send ( ) is as follows: 

] send(self, to, subject=' A/one' , message='A/one' , attachments=l, 
z cc=l, bcc=l, reply_to=l, encoding='utf-8' ,headers={}) 

Note, to, cc, and bcc each take a list of email addresses. 

headers is dictionary of headers to refine the headers just before sending the 
email. For example: 

1 headers = {'Return-Path' : 'bounces@example.org'} 

Following are some additional examples demonstrating the use of 
mail . send( ). 


8.2.1 Simple text email 


1 mail. send(' you@example.com' , 

2 'Message subject', 

3 'Plain text body of the message') 


8.2.2 HTML emails 


1 mail. send(' you@example.com' , 

2 'Message subject', 

3 ' <html>html body</html>' ) 

If the email body starts with <html> and ends with </html>, it will be sent as 
a HTML email. 

8.2.3 Combining text and HTML emails 

The email message can be a tuple (text, html): 

1 mail. send(' you@example.com' , 

2 'Message subject', 

3 ('Plain text body', ' <html>html body</html>' )) 
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8.2.4 cc an d bcc emails 



S.2.5 Attachments 


mail. send ( 'you@example. com' , 
'Message subject' , 

' <htmlximg src="cid: photo" /></html>' , 
attachments = Mail. Attachment ( ' /path/to/photo. jpg' , content_id=' photo' ) ) 


8.2.6 Multiple attachments 



8.3 Sending SMS messages 

Sending SMS messages from a web2py application requires a third party 
service that can relay the messages to the receiver. Usually this is not a free 
service, but it differs from country to country. We have tried a few of these 
services with little success. Phone companies block emails originating from 
these services since they are eventually used as a source of spam. 

A better way is to use the phone companies themselves to relay the SMS. 

Each phone company has an email address uniquely associated with every 

cell-phone number, so SMS messages can be sent as emails to the phone 

number. web2py comes with a module to help in this process: 

from gluon.contrib.sms_utils import SMSCODES, sms_email 
email = sms_email( ' I (111) 111-1111' ,'T-Mobile USA (tmail)') 
mail. sent (to=email, subject=' test ' , message^' test ' ) 

SMSCODES is a dictionary that maps names of major phone companies to 
the email address postfix. The sms_email function takes a phone number (as 
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a string) and the name of a phone company and returns the email address of 
the phone. 

8.4 Using the template system to generate messages 

It is possible to use the template system to generate emails. For example, 
consider the database table 
1 db.define_table( 'person ' , Field (' name ') ) 

where you want to send to every person in the database the following 
message, stored in a view file "message.html": 

1 Dear {{=person . name}}, 

2 You have won the second prize, a set of steak knives. 

You can achieve this in the following way 

1 for person in db(db. person) . select () : 

2 context = diet (person=person) 

3 message = response. render( 'message.html ' , context) 

4 mail. send(to=[ ' who@example . com' ] , 

5 subject='Wone' , 

6 message=message) 

Most of the work is done in the statement 

] response. render( 'message. html' , context) 

It renders the view "message.html" with the variables defined in the 
dictionary "context", and it returns a string with the rendered email text. 
The context is a dictionary that contains variables that will be visible to the 
template file. 

If the message starts with <html> and ends with </html>, the email will be an 
HTML email. 

Note, if you want to include a link back to your website in an HTML email, 
you can use the URL function. However, by default, the URL function generates 
a relative URL, which will not work from an email. To generate an absolute 
URL, you need to specify the scheme and host arguments to the URL function. 
For example: 
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<a h ref =" {{=URL (. .. , scheme=True, host-True)}} ">Click here</a> 


or 

<a h ref =" {{=URL (. .. , scheme^' http' , host=' www. site, com' )}}">C\-ick here</a> 

The same mechanism that is used to generate email text can also be used to 
generate SMS messages or any other type of message based on a template. 

8.5 Sending messages using a background task 

The operation of sending an email message can take up to several seconds 
because of the need to log into and communicate with a potentially remote 
SMTP server. To keep the user from having to wait for the send operation 
to complete, it is sometimes desirable to queue the email to be sent at a later 
time via a background task. As described in Chapter 4, this can be done by 
setting up a homemade task queue or using the web2py scheduler. Here we 
provide an example using a homemade task queue. 

First, in a model file within our application, we set up a database model to 
store our email queue: 



From a controller, we can then enqueue messages to be sent by: 

] db. queue. insert (status=' pending ' , 

z emails ' you@example . com' , 

3 subject^' test' , 

4 message^' test ' ) 

Next, we need a background processing script that reads the queue and sends 
the emails: 

1 # in file /app/private/wail_queue.py 

2 import time 

3 while True: 

4 rows = db(db. queue. status=='pending' ) .select ( ) 
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for 

row in rows: 




if 

mail . send(to: 
subject=row 
message=row 

=row. email, 
.subject, 
.message) : 




row. update. 

record(status= 

'sent' ) 


else: 





row. update. 

record(status= 

'failed') 


db. 

commit ( ) 



time. sleep (60) # check every minute 


Finally, as described in Chapter 4, we need to run the mail_queue.py script 

as if it were inside a controller in our app: 

python web2py.py -S app -M -N -R applications/app/private/mail_queue.py 

where -S app tells web2py to run "mail_queue.py" as "app", -M tells web2py 
to execute models, and - N tells web2py not to run cron. 

Here we assume that the mail object referenced in "mail_queue.py" is defined 
in a model file in our app and is therefore available in the "mail_queue.py" 
script because of the -M option. Also notice that it is important to commit 
any change as soon as possible in order not to lock the database to other 
concurrent processes. 

As noted in Chapter 4, this type of background process should not be 
executed via cron (except perhaps for cron ©reboot) because you need to 
be sure that no more than one instance is running at the same time. 

Note, one drawback to sending email via a background process is that it 
makes it difficult to provide feedback to the user in case the email fails. If 
email is sent directly from the controller action, you can catch any errors 
and immediately return an error message to the user. With a background 
process, however, the email is sent asynchronously, after the controller action 
has already returned its response, so it becomes more complex to notify the 
user of a failure. 


9 

Access Control 


web2py includes a powerful and customizable Role Based Access Control 
mechanism (RBAC). 

Here is a definition from Wikipedia: 

"Role-Based Access Control (RBAC) is an approach to restricting system 
access to authorized users. It is a newer alternative approach to mandatory 
access control (MAC) and discretionary access control (DAC). RBAC is 
sometimes referred to as role-based security. 

RBAC is a policy neutral and flexible access control technology sufficiently 
powerful to simulate DAC and MAC. Conversely MAC can simulate RBAC 
if the role graph is restricted to a tree rather than a partially ordered set. 

Prior to the development of RBAC, MAC and DAC were considered to be 
the only known models for access control: if a model was not MAC, it was 
considered to be a DAC model, and vice versa. Research in the late 1990s 
demonstrated that RBAC falls in neither category. 

Within an organization, roles are created for various job functions. The 
permissions to perform certain operations are assigned to specific roles. 
Members of staff (or other system users) are assigned particular roles, and 
through those role assignments acquire the permissions to perform particular 
system functions. Unlike context-based access control (CBAC), RBAC does 
not look at the message context (such as a connection's source). 
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Since users are not assigned permissions directly, but only acquire them 
through their role (or roles), management of individual user rights becomes 
a matter of simply assigning appropriate roles to the user; this simplifies 
common operations, such as adding a user, or changing a user 's department. 

RBAC differs from access control lists (ACLs) used in traditional 
discretionary access control systems in that it assigns permissions to specific 
operations with meaning in the organization, rather than to low level data 
objects. For example, an access control list could be used to grant or deny 
write access to a particular system file, but it would not dictate how that file 
could be changed." 

The web2py class that implements RBAC is called Auth. 

Auth needs (and defines) the following tables: 

• auth_user stores users' name, email address, password, and status 
(registration pending, accepted, blocked) 

• auth_group stores groups or roles for users in a many-to-many structure. 
By default, each user is in its own group, but a user can be in multiple 
groups, and each group can contain multiple users. A group is identified 
by a role and a description. 

• auth_membership links users and groups in a many-to-many structure. 

• auth_permission links groups and permissions. A permission is identified 
by a name and, optionally, a table and a record. For example, members 
of a certain group can have "update" permissions on a specific record of a 
specific table. 

• auth_event logs changes in the other tables and successful access via CRUD 
to objects controlled by the RBAC. 

In principle, there is no restriction on the names of the roles and the 
names of the permissions; the developer can create them to fix the roles 
and permissions in the organization. Once they have been created, web2py 
provides an API to check if a user is logged in, if a user is a member of a 
given group, and /or if the user is a member of any group that has a given 
required permission. web2py also provides decorators to restrict access to 
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any function based on login, membership and permissions. web2py also 
understands some specific permissions, i.e., those that have a name that 
correspond to the CRUD methods (create, read, update, delete) and can 
enforce them automatically without the need to use decorators. 

In this chapter, we are going to discuss different parts of RBAC one by one. 


9.2 Authentication 

In order to use RBAC, users need to be identified. This means that they need 
to register (or be registered) and log in. 

Auth provides multiple login methods. The default one consists of 
identifying users based on the local auth_user table. Alternatively, it can log in 
users against third-party authentication systems and single sign on providers 
such as Google, PAM, LDAP, Facebook, Linkedln, Dropbox, OpenID, OAuth, 
etc.. 

To start using Auth, you need at least this code in a model file, which is 
also provided with the web2py "welcome" application and assumes a db 
connection object: 


from 

gluon. tools 

impc 

irt , 

Auth 





auth 

= Auth(db, 

hmac_ 

key; 

=Auth 

.get. 

.or. 

.create. 

-keyO) 

auth 

.define_tabl 

es() 








The password field of the db.auth_user table defaults to a CRY PT validator, which 
needs and hmac_key. The Auth.get_or_create_key( ) is a function that read the 
hmac kay from a file "private/auth.key" within the application folder. If the file 
does not exist it creates a random hmac_key. If multiple apps share the same 
auth database, make sure they also use the same hmac_key. 

By default, web2py uses email for login. If instead you want to log in using 
username set auth.define_tables(username=True) 

If multiple apps share the same auth database you may want to disable 
migrations: auth. define- tables (migrate=False). 

To expose Auth, you also need the following function in a controller (for 
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example in "default.py"): 

1 def user(): return dict(form=auth( ) ) 

The auth object and the user action are already defined in the scaffolding 
application. 

web2py also includes a sample view "welcome/views/default/user.html" to 

render this function properly that looks like this: 

] {{extend 'layout.html'}} 

z <h2>{{=T( request .args(0) . replace! ' ',' '(.capitalize)) )}}</h2> 

3 <div id="web2py_user_form"> 

4 {{=form}} 

5 {{if request .args(0)==' login' : }} 

6 {{if not 'register' in auth. settings. actions_disabled: }} 

7 <br/xa bref="{{=URL(args= : register' )}}">reqister</a> 
s {{pass}} 

i) {{if not ' request_reset_password' in auth . settings. actions_disabled:}} 

10 <br/> 

u <a href="{" {=URL(args=' request_reset_password' )}}">\ost password</a> 

{{pass}} 
.3 {{pass}} 
14 </div> 

Notice that this function simply displays a form and therefore it can be 
customized using normal custom form syntax. The only caveat is that the 
form displayed by form=auth( ) depends on request .args(Q); therefore, if you 
replace the default auth() login form with a custom login form, you may 
need an if statement like this in the view: 
1 {{if request .args(0)==' login' :}}.. .custom login form. . .{{pass}} 

The controller above exposes multiple actions: 


http://. 

, ./[app]/default/user/ register 

http://. 

. ./[app]/default/user/login 

http://. 

, ./[app]/default/user/logout 

http://. 

, ./[app]/default/user/profile 

http://. 

. . /[app] /default/use r/ change. password 

http://. 

, ./[app]/default/user/verify_email 

http://. 

. ./ [app] /default/user/ ret rieve_username 

http://. 

, . / [app] /default/user/ request- reset- pas sword 

http://. 

, ./ [app] /default/user/ reset_password 

http://. 

, ./[app] /default/user/impersonate 

http://. 

, ./[app]/default/user/groups 

http://. 

. ./[app]/default/user/not_authorized 
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• register allows users to register. It is integrated with CAPTCHA, although 
this is disabled by default. 

• login allows users who are registered to log in (if the registration is 
verified or does not require verification, if it has been approved or does 
not require approval, and if it has not been blocked). 

• logout does what you would expect but also, as the other methods, logs 
the event and can be used to trigger some event. 

• profile allows users to edit their profile, i.e. the content of the auth_user 
table. Notice that this table does not have a fixed structure and can be 
customized. 

• change_password allows users to change their password in a fail-safe way. 

• verify_email. If email verification is turned on, then visitors, upon 
registration, receive an email with a link to verify their email information. 
The link points to this action. 

• retrieve_username. By default, Auth uses email and password for login, 
but it can, optionally, use username instead of email. In this latter case, if 
a user forgets his/her username, the retrieve_username method allows the 
user to type the email address and retrieve the username by email. 

• request_reset_password. Allows users who forgot their password to 
request a new password. They will get a confirmation email pointing 
to reset_password. 

• impersonate allows a user to "impersonate" another user. This is 
important for debugging and for support purposes, request .args[0] is 
the id of the user to be impersonated. This is only allowed if the logged in 
user has_permission( 'impersonate' , db.auth_user, user_id). 

• groups lists the groups of which the current logged in user is a member. 

• not_authorized displays an error message when the visitor tried to do 
something that he/she is not authorized to do 

• navbar is a helper that generates a bar with login/register/etc. links. 
Logout, profile, change_password, impersonate, and groups require login. 
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By default they are all exposed, but it is possible to restrict access to only 
some of these actions. 

All of the methods above can be extended or replaced by subclassing Auth. 

All of the methods above can be used in separate actions. For example: 

1 def mylogin(): return diet (form=auth.login( ) ) 

z def myregister( ) : return diet (form=auth. register! ) ) 

3 def myprofile(): return diet (form=auth.profile( ) ) 

4 ... 

To restrict access to functions to only logged in visitors, decorate the function 
as in the following example 


1 @auth. requires_login( ) 

2 def hello () : 

3 return diet (message='helZo %(first_name)s' % auth. user) 

Any function can be decorated, not just exposed actions. Of course this is 
still only a very simple example of access control. More complex examples 
will be discussed later. 

auth. user contains a copy of the db.auth_user records for the current logged 
in user or None otherwise. There is also a auth. use r_id which is the same as 
auth. user. id (i.e. the id of the current logger in user) or None. 


9.1.1 Restrictions on registration 

If you want to allow visitors to register but not to log in until registration has 

been approved by the administrator: 

auth. settings . registration_requires_approval = True 

You can approve a registration via the appadmin interface. Look into the 
table auth_user. Pending registrations have a registration-key field set to 
"pending". A registration is approved when this field is set to blank. 

Via the appadmin interface, you can also block a user from logging in. Locate 
the user in the table auth_user and set the registration-key to "blocked", 
"blocked" users are not allowed to log in. Notice that this will prevent a 
visitor from logging in but it will not force a visitor who is already logged in 
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to log out. The word "disabled" may be used instead of "blocked" if preferred, 
with exactly the same behavior. 

You can also block access to the "register" page completely with this 
statement: 

auth. settings .actions_disabled .append) ' register' ) 

If you want to allow people to register and automatically log them in after 
registration but still want to send an email for verification so that they cannot 
login again after logout, unless they completed the instructions in the email, 
you can accomplish it as follows: 

auth. settings . registration_requires_approval = True 
auth. settings .login_after_registration = True 

Other methods of Auth can be restricted in the same way. 


9.1.2 Integration with OpenID, Facebook, etc. 

You can use the web2py Role Base Access Control and authenticate with 
other services like OpenID, Facebook, Linkedln, Google, Dropbox, MySpace, 
Flickr, etc. The easiest way is to use Janrain Engage (formerly RPX) 
0anrain.com). 

Dropbox is discussed as a special case in Chapter 14 since it allows more than 
just login, it also provides storage services for the logged in users. 

Janrain Engage is a service that provides middleware authentication. You 
can register with Janrain.com, register a domain (the name of your app) and 
set of URLs you will be using, and they will provide you with an API key. 

Now edit the model of your web2py application and place the following lines 
somewhere after the definition of the auth object : 

1 from gluon.contrib.login_methods. rpx_account import RPXAccount 

2 auth. settings .actions_disabled=[ ' register' , ' change_password' , ' 

request-reset-password' ] 

3 auth. settings .login_form = RPXAccount (request, 

4 api_key=' . . . ' , 

5 domain=' 

6 url = "http:/ /localhost:8Q0B/%s/default/user/login" % request. application) 
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The first line imports the new login method, the second line disables local 
registration, and the third line asks web2py to use the RPX login method. 
You must insert your own api_key provided by Janrain.com, the domain you 
choose upon registration and the external url of your login page. 


* I * + Wfittp://127.0.0.LB000/welcome/defaullf r user/lo9in 


welcome 


Login 

Sign in using your account with 


AOLfctf- 

Google 

myf^penlD 

OpenID 

flickr 


When a new user logins for the first time, web2py creates a new db.auth_user 
record associated to the user. It will use the registration-id field to store 
a unique id for the user. Most authentication methods will also provide 
a username, email, first_name and last_name but that is not guaranteed. 
Which fields are provided depends on the login method selected by the user. 
If the same user logs in twice using different authentication mechanisms 
(for example once with OpenID and once with Facebook), Janrain may not 
recognize his/her as the same user and issue different registration_id. 

You can customize the mapping between the data provided by Janrain and 
the data stored in db.auth_user. Here is an example for Facebook: 

auth. settings .login_form. mappings. Facebook = lambda profile:\ 
diet ( registration_id = profile[ "identifier" ] , 
username = profile["preferrecfl/sername"] , 
email = p rofile ["email"] , 
first_name = profile["name" ] [ "givenName"] , 
last_name = profile["name"] [ "familyName"] ) 


The keys in the dictionary are fields in db.auth_user and the values are data 
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entries in the profile object provided by Janrain. Look at the online Janrain 
documentation for details on the latter. 

Janrain will also keep statistics about your users' login. 

This login form is fully integrated with web2py Role Based Access Control 
and you can still create groups, make users members of groups, assign 
permissions, block users, etc. 

Janrain's free Basic service allows up to 2500 unique registered users to sign in 
annually. Accommodating more users requires an upgrade to one of their paid 
service tiers. 

If you prefer not to use Janrain and want to use a different login method (LDAP, 
PAM, Google, OpenID, OAuth/Facebook, Linkedln, etc.) you can do so. The 
API to do so is described later in the chapter. 


9.1.3 CAPTCHA and reCAPTCHA 

To prevent spammers and bots registering on your site, you may require a 
registration CAPTCHA. web2py supports reCAPTCHA [74] out of the box. 
This is because reCAPTCHA is very well designed, free, accessible (it can 
read the words to the visitors), easy to set up, and does not require installing 
any third-party libraries. 

This is what you need to do to use reCAPTCHA: 

• Register with reCAPTCHA [74] and obtain a (PUBLICJCEY, 
PRIVATE_KEY) couple for your account. These are just two strings. 

• Append the following code to your model after the auth object is defined: 

1 from gluon. tools import Recaptcha 

2 auth. settings . captcha = Recaptcha(request, 

3 , PUBLIC_KEY' , 'PRIVATE-KEY') 

reCAPTCHA may not work if you access the web site as Tocalhost' or 
'127.0.0.1', because it is registered to work with publicly visible web sites 
only. 

The Recaptcha constructor takes some optional arguments: 
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Recaptcha( . . . , use_ssl=True, error_message=' invalid' , \abe\-' Verify: ' , options='') 

Notice that use_ssl=False by default. 

options may be a configuration string, e.g. options="theme: 'white' , 
lang : 'f r' " 

More details: reCAPTCHA [? ] and customizing [? ]. 

If you do not want to use reCAPTCHA, look into the definition of the 
Recaptcha class in "gluon/tools.py", since it is easy to use other CAPTCHA 
systems. 

Notice that Recaptcha is just a helper that extends DIV. It generates a dummy 
field that validates using the reCaptcha service and, therefore, it can be used 
in any form, including used defined FORMs: 

form = FORM(INPUT(. . . ) ,Recaptcha( . . . ) ,INPUT(_type='submit ' ) ) 

You can use it in all types of SQLFORM by injection: 

form = SQLFORM(. . .) or SQLFORM. factory) ... ) 

form. element! ' table' ) .insert ( -1,TR( ' ' , Recaptcha ( ...),'') ) 


9.1.4 Customizing Auth 

The call to 

auth.define_tables( ) 

defines all Auth tables that have not been defined already. This means that if 
you wish to do so, you can define your own auth_user table. 

There are a number of ways to customize auth. The simplest way is to add 
extra fields: 
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You can declare extra fields not just for table "auth_user" but also for other 
"auth_" tables. Using extra_fields is the recommended way as it will not 
break any internal mechanism. 

Another way to do (for experts!) consists of defining your auth tables youself. 
If a table is declared before auth.define_tables() it is used instead of the 
default one. Here is how to do it: 



# after auth = Auth(db) 


3 

db.define_table( 

auth. settings. table_ use r_ name, 


4 

Field (' firsts name' , length=128, default=' ' ) , 


5 

Field (' last_name' , length=128, default='') : 


6 

Field (' emai 1' , length=128, default^ 11 , unique=True) , # 

required 

7 

Field ( 'password' , 'password', length=512, # 

required 

8 

readable=False, label= ' Password ' ) , 
Field ( 'address ' ) , 
Field ('city'), 
Field ( 'zip' ) , 


12 

Field ( 'phone' ) , 


13 

Field ( ' regis t rati on_ key ' , length=512, # 

required 

14 

writable=False, readable=False, default^ 11 ), 


15 

Field (' reset-password-key ' , length=512, # 

required 

16 

writable=False, readable=False, default^ 11 ), 


*7 

Field (' registration-id' , length=512, # 

required 

18 

writable=False, readable=False, default^ 11 )) 


19 

# do not forget validators 


;; 

custom_auth_table = db[auth. settings. table_user_name] # get 
custom_auth_table.first_name. requires = \ 

the custom-auth-table 

23 

IS_NOT_EMPTY(error_message=auth. messages .is_empty) 


^4 

custom_auth_table.last_name. requires = \ 


25 

IS_NOT_EHPTY(error_message=auth. messages .is_empty) 


26 

custom_auth_table. password, requires = [IS_STRONG( ) , CRYPTO] 

27 

custom_auth_table. email. requires = [ 


28 

IS_EMAIL(error_message=auth . mes sages. invalid_email) , 


29 

IS_NOT_IN_DB(db, custom_auth_table. email ) ] 


30 

auth. settings .table_user = custom_auth_table # tell auth tc 

1 use custom-auth-table 

32 

33 

# before auth.define-tablesO 



You can add any field you wish, and you can change validators but you 
cannot remove the fields marked as "required" in this example. 
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It is important to make "password", "registration_key", "reset_password_key" 
and "registration_id" fields readable=False and writable=False, since a visitor 
must not be allowed to tamper with them. 

If you add a field called "username", it will be used in place of "email" for 
login. If you do, you will need to add a validator as well: 

auth_table. username. requires = IS_NOT_IN_DB(db : auth_table. username) 


9.1.5 Renaming Auth tables 

The actual names of the Auth tables are stored in 



The names of the table can be changed by reassigning the above variables 
after the auth object is defined and before the Auth tables are defined. For 
example: 


auth = Auth(db) 


auth. settings . table_user_name = 

■ 'person' 

#. . . 


auth.define_tables( ) 



The actual tables can also be referenced, independently of their actual names, 

by 


1 auth. settings .table_user 

2 auth. settings . table_group 

3 auth. settings . table_membership 

4 auth. settings . table_permission 

5 auth. settings .table_event 


9.2.6 Other login methods and login forms 

Auth provides multiple login methods and hooks to create new login 
methods. Each supported login method corresponds to a file in the folder 
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] gluon/contrib/login_methods/ 

Refer to the documentation in the files themselves for each login method, but 
here are some examples. 

First of all, we need to make a distinction between two types of alternate 
login methods: 

• login methods that use a web2py login form (although the credentials are 
verified outside web2py). An example is LDAP. 

• login methods that require an external single-sign-on form (an example is 
Google and Facebook). 

In the latter case, web2py never gets the login credentials, only a 
login token issued by the service provider. The token is stored in 
db.auth_user. registration_id. 

Let's consider examples of the first case: 

Basic 

Let's say you have an authentication service, for example at the url 

1 https://basic.example.com 

that accepts basic access authentication. That means the server accepts HTTP 
requests with a header of the form: 

, GET /index. html HTTP/1.0 

2 Host: basic.example.com 

3 Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ== 

where the latter string is the base64 encoding of the string 
username:password. The service responds 200 OK if the user is authorized 
and 400, 401, 402, 403 or 404 otherwise. 

You want to enter username and password using the standard Auth login 
form and verify the credentials against such a service. All you need to do is 
add the following code to your application 


from gluon. contrib.login_methods.basic_auth import basic_auth 
auth . settings . login_methods . append ( 

basic_auth( ' https :// basic . example . com' ) ) 
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Notice that auth. settings. login_methods is a list of authentication methods 
that are executed sequentially. By default it is set to 

1 auth. settings .login_methods = [auth] 

When an alternate method is appended, for example basic_auth, Auth first 
tries to log in the visitor based on the content of auth_user, and when this 
fails, it tries the next method in the list. If a method succeeds in logging in the 
visitor, and if auth. settings. login_methods [0]==auth, Auth takes the following 
actions: 

• if the user does not exist in auth_user, a new user is created and the 
username/ email and passwords are stored. 

• if the user does exist in auth_user but the new accepted password does 
not match the old stored password, the old password is replaced with the 
new one (notice that passwords are always stored hashed unless specified 
otherwise). 

If you do not wish to store the new password in auth_user, then it is sufficient 
to change the order of login methods, or remove auth from the list. For 
example: 

1 from gluon. contrib."Login_methods.basic_auth import basic_auth 

2 auth. settings .login_methods = \ 

3 [basic_auth( ' https :// basic . example . com' ) ] 

The same applies for any other login method described here. 

SMTP and Gmail 

You can verify login credentials using a remote SMTP server, for example 
Gmail; i.e., you log the user in if the email and password they provide are 
valid credentials to access the Gmail SMTP server (smtp.gmail.com: 587). All 
that is needed is the following code: 

i from gluon. cont rib. login_methods.email_auth import email_auth 

2 auth . settings . login_methods . append ( 

3 email_auth( "smtp. gmail. com: 587" , "@gmail.com")) 

The first argument of email_auth is the address:port of the SMTP server. The 
second argument is the email domain. 
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This works with any SMTP server that requires TLS authentication. 

PAM 

Authentication using Pluggable Authentication Modules (PAM) works as in 
the previous cases. It allows web2py to authenticate users using the operating 
system accounts: 

from gluon. contrib."login_methods.pam_auth import pam_auth 
auth. settings .login_methods . append (pam_auth( ) ) 

LDAP 

Authentication using LDAP works very much as in the previous cases. 

To use LDAP login with MS Active Directory: 

from gluon. contrib.login_methods.ldap_auth import ldap_auth 
auth. settings .login_methods . append (ldap_auth(mode=' ad' , 

server='my. domain, controller' , 

base_dn=' ou=Users , dc=domain , dc=com' ) ) 

To use LDAP login with Lotus Notes and Domino: 

auth. settings .login_methods . append (ldap_auth(mode=' domino' , 
server='my. domino, server' ) ) 

To use LDAP login with OpenLDAP (with UID): 

auth. settings .login_methods . append (ldap_auth(server=' my. Idap. server' , 
base_dn=' ou=Users , dc=domain , dc=com' ) ) 

To use LDAP login with OpenLDAP (with CN): 

auth. settings .login_methods . append CLdap_auth(mode=' en ' , 

server^' my. Idap. server' , base_dn= ' ou=Users, dc=domain, dc=com' ) ) 

Google App Engine 

Authentication using Google when running on Google App Engine requires 
skipping the web2py login form, being redirected to the Google login page, 
and back upon success. Because the behavior is different than in the previous 
examples, the API is a little different. 

from gluon. contrib.login_methods.gae_google_login import GaeGoogleAccount 
auth. settings .login_form = GaeGoogleAccount ( ) 
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OpenID 

We have previously discussed integration with Janrain (which has OpenID 
support) and that is the easiest way to use OpenID. Yet sometimes you do 
not want to rely on a third party service and you want to access the OpenID 
provider directly from the consumer (your app). 

Here is an example: 

from gluon. contrib.login_methods.openid_auth import OpenlDAuth 
auth. settings .login_form = OpenlDAuth(auth) 

OpenlDAuth requires the python-openid module to be installed separately. 
Under the hood, this login method defines the following table: 

db.define_table( 'alt-logins' , 

Field ( 'use rname' , length=512, default=' ' ) , 

Field (' type ' , length =128, default= ' openid ' , readable=False) , 

Field( 'user' , self .table_user, readable=False) ) 

which stores the openid usernames for each user. If you want to display the 
openids for the current logged in user: 

{{=auth.settings.login_form.list_user_openids( )}} 

OAutti2.o and Facebook 

We have previously discussed integration with Janrain (which has Facebook 
support), yet sometimes you do not want to rely on a third party service 
and you want to access a OAuth2.o provider directly; for example, Facebook. 
Here is how: 

from gluon. contrib.login_methods.oauth2CLaccount import OAuthAccount 
auth. sett ings .login_form=OAuthAccount(YOUR_CLIENT_ID,YOUR_CLIENT_SECRET) 

Things get a little more complex if you want to use Facebook OAuth2.o to 
login into a specific Facebook app to access its API, instead of your own app. 
Here is an example for accessing the Facebook Graph API. 

First of all you must install the Facebook Python SDK. 

Second, you need the following code in your model: 

# import required modules 
from facebook import GraphAPI 
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from gluon. contrib.login_methods.oauth20_account import OAuthAccount 

# extend the OAUthAccount class 
class FaceBookAccount(OAuthAccount) : 

OAuth impl for Facebook 

AUTH_URL=" https :/ /graph . facebook. com/ oauth/ authorize" 
T0KEN_URL="/7rrps://graph. facebook. com/ oauth/ access-token" 

def init (self, g) : 

OAuthAccount init (self, g, 

YOUR_CLIENT_ID, 
YOUR_CLIENT_SECRET, 
self .AUTH_URL, 
self .T0KEN_URL) 
self. graph = None 
# override function that fetches user info 
def get_user(self ) : 

"Returns the user using the Graph API" 
if not self .accessToken( ) : 

return None 
if not self. graph: 

self. graph = GraphAPI( (self .accessToken( ) ) ) 
try: 

user = self .graph .get_object( "me" ) 
return dict(first_name = user[ ' first-name' ] , 
last_name = user[ ' last-name' ] , 
username = user[ 'id']) 
except GraphAPIError: 

self .session. token = None 
self. graph = None 
return None 

# use the above class to build a new login form 
auth. settings .login_form=FaceBookAccount( ) 

Linkedln 

We have previously discussed integration with Janrain (which has Linkedln 
support) and that is the easiest way to use OAuth. Yet sometime you do not 
want to rely on a third party service or you may want to access Linkedln 
directly to get more information than Janrain provides. 


Here is an example: 


1 from gluon. contrib.login_methods.linkedin_account import LinkedlnAccount 

2 auth . settings . login_f orm=LinkedInAccount ( request , KEY, SECRET, RETURN_URL) 

LinkedlnAccount requires the "python-linkedin" module installed separately. 
X509 
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You can also login by passing to the page an X509 certificate and your 
credential will be extracted from the certificate. This requires M2Crypto 
installed from 

] http://chandlerproject.org/bin/view/Projects/MeTooCrypto 

Once you have M2Cryption installed you can do: 

1 from gluon.contrib.login_methods.x509_auth import X509Account 

2 auth. settings .actions_disabled=[ ' register' , ' change-password' , ' 

request_reset_password' ] 

3 auth. settings .login_form = X509Account( ) 

You can now authenticate into web2py passing your X509 certificate. How 
to do this is browser dependant but probably you are more likely to use 
certificates for web services. In this case you can use for example cURL to try 
out your authentication: 

1 curl -d "firstName=John&lastName=Smith" -G -v --key private. key \ 

2 --cert server. crt https ://example/app/default/user/profile 

This works out of the box with Rocket (the web2py built-in web server) but 
you may need some extra configuration work on the web server side if you 
are using a different web server. In particular you need to tell your web server 
where the certificates are located on local host and that it needs to verify 
certificates coming from the clients. How to do it is web server dependent 
and therefore omitted here. 

Multiple login forms 

Some login methods modify the login_form, some do not. When they do 
that, they may not be able to coexist. Yet some coexist by providing multiple 
login forms in the same page. web2py provides a way to do it. Here is an 
example mixing normal login (auth) and RPX login (janrain.com): 

1 from gluon. contrib.login_methods.extended_login_form import ExtendedLoginForm 

2 other_form = RPXAccount( request, api_key=' . . . ' , domain=' . . . ' , url='...') 

3 auth. settings .login_form = Extendedl_oginForm(request, 

4 auth, other_form, signals=[ ' token '] ) 

If signals are set and a parameter in request matches any signals, it will 
return the call of other_form.login_form instead. other_form can handle some 
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particular situations, for example, multiple steps of OpenID login inside 
other_form. login_form. 

Otherwise it will render the normal login form together with the other_form. 


9.2 Mail and Auth 

One can define a mailer with 



or simply use the mailer provided by auth: 



You need to replace the mail. settings with the proper parameters for your 
SMTP server. Set mail. settings. login=False if the SMTP server does not 
require authentication. 

You can read more about web2py API for emails and email configuration in 
Chapter 8. Here we limit the discussion to the interaction between Mail and 
Auth. 

In Auth, by default, email verification is disabled. To enable email, append 
the following lines in the model where auth is defined: 

auth. settings . registration_requires_verification = False 
auth. settings . registration_requires_approval = False 
auth. settings . reset_password_requires_verification = True 
auth. messages . verify_email = 'Click on the link http://' + \ 

request .env.http_host + \ 

URL( r=request,c='c/efault' ,f='user' ,args=[ ' verify_email' ] ) + \ 

' /%(key)s to verify your email' 
auth. messages . reset_password = 'Click on the link http://' + \ 

request .env.http_host + \ 

URL( r=request, c=' default' , f=' user' ,args=[ ' reset_password' ] ) + \ 

'/%(key)s to reset your password' 
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You need to replace the string 
'Click on the link . . . ' 

in auth. messages. verify_email with the proper complete URL of the action 
verify_email. This is necessary because web2py may be installed behind a 
proxy and it cannot determine its own public URLs with absolute certainty. 

9.3 Authorization 

Once a new user is registered, a new group is created to contain the user. 
The role of the new user is conventionally "user_[id]" where [id] is the id of 
the newly created id. The creation of the group can be disabled with 
auth. settings . create_user_groups = False 

although we do not suggest doing so. 

Users have membership in groups. Each group is identified by a name/role. 
Groups have permissions. Users have permissions because of the groups 
they belong to. 

You can create groups, give membership and permissions via appadmin or 
programmatically using the following methods: 

auth.add_group( 'role' , 'description' ) 

returns the id of the newly created group. 

auth. del_g roup (group_id) 

deletes the group with group_id. 
auth.del_group(auth.id_group( 'user. 7' ) ) 

deletes the group with role "user_7", i.e., the group uniquely associated to 

user number 7. 

auth. user_g roup ( use r_id) 

returns the id of the group uniquely associated to the user identified by 
user_id. 
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auth.add_membership(group_id : user_id) 


gives user_ id membership of the group group_id. If the user_id is not 
specified, then web2py assumes the current logged-in user. 

auth.del_membership(group_id : user_id) 

revokes user_id membership of the group group_id. If the user_id is not 
specified, then web2py assumes the current logged-in user. 

auth.has_membership(group_id : user_id, role) 

checks whether user_id has membership of the group group_id or the group 
with the specified role. Only group.id or role should be passed to the 
function, not both. If the user_id is not specified, then web2py assumes the 
current logged-in user. 

auth.add_permission(group_id : 'name', 'object', record_id) 

gives permission "name" (user defined) on the object "object" (also user 
defined) to members of the group group_id. If "object" is a tablename then 
the permission can refer to the entire table by setting record-id to a value of 
zero, or the permission can refer to a specific record by specifying a record_id 
value greater than zero. When giving permissions on tables, it is common to 
use a permission name in the set ('create', 'read', 'update', 'delete', 'select') 
since these permissions are understood and can be enforced by CRUD. 

If group_ id is zero, web2py uses the group uniquely associated to the current 
logged-in user. 

You can also use auth.id_group(role=" . . . ") to get the id of a group given its 
name. 


auth.del_permission(group_id : 'name' , 

'object' , 

record_id) 

revokes the permission. 

auth.has_ permission! 'name' , 'object' , 

record_id, 

user_id) 


checks whether the user identified by user_id has membership in a group 
with the requested permission. 
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rows = db(auth .accessible_query( ' read' , db.mytable, user_id))\ 
. select ( db . my table . ALL) 

returns all rows of table "mytable" that user user_id has "read" permission 
on. If the user_id is not specified, then web2py assumes the current logged- 
in user. The accessible_query( . . . ) can be combined with other queries to 
make more complex ones. accessible- que ry(...) is the only Auth method to 
require a JOIN, so it does not work on the Google App Engine. 


Assuming the following definitions: 


, >» from gluon. tools import Auth 

2 >» auth = Auth(db) 

3 >» auth.define_tables( ) 

4 >» secrets = db.define_table( 'document' , Field (' body ') ) 

5 >» james_bond = db.auth_user.insert(first_name=' James ' , 

6 last_name=' Bond' ) 

Here is an example: 

1 >» doc_id = db. document, insert (body = 'top secret') 

2 >» agents = auth. add_group( role = 'Secret Agent') 

3 >» auth.add_membership(agents : james_bond) 

4 >» auth.add_permission(agents : 'read', secrets) 

5 >» print auth .has_permission( ' read' , secrets, doc_id, james_bond) 

6 True 

- >» print auth .has_permission( 'update' , secrets, doc_id, james_bond) 

8 False 


9.3.1 Decorators 

The most common way to check permission is not by explicit calls to the 
above methods, but by decorating functions so that permissions are checked 
relative to the logged-in visitor. Here are some examples: 

1 def function_one( ) : 

2 return 'this is a public function' 

3 

4 @auth. requires_login( ) 

5 def f unction_ two( ) : 

6 return 'this requires login' 
7 

s @auth. requires_membership( 'agents ' ) 
9 def function_three( ) : 
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10 

return 'you are a secret agent' 



@auth. requires^pe mission ( ' read' , secrets) 
def function_four( ) : 


14 

return 'you can read secret documents' 


15 
16 

@auth. requires_permission( 'delete' , 'any file') 


*7 

def function_five( ) : 


18 

import os 


!9 

for file in os.listdirf ' ./' ) : 


20 

os . unlink(file) 


21 

return 'all files deleted' 


22 
23 

@auth. requires (auth.user_id==l or request . client== '127.0. 0.1 ' , 

requires_login=True) 

^4 

def function_six( ) : 


25 

return 'you can read secret documents' 


26 
27 

@auth. requires- pe mission ( 'add' , 'number' ) 


28 

def add(a, b) : 


29 

return a + b 


3° 
31 

def function_seven( ) : 


32 

return add(3, 4) 



The condition argument of @auth. requires(condition) can be a callable. 
@auth . requires also takes an optional argument requires_login which defaults 
to True. If set to False, it does not require login before evaluating the condition 
as true /false. The condition can be a boolean value or a function evaluating 
to boolean. 

Note that access to all functions apart from the first one is restricted based 
on permissions that the visitor may or may not have. 

If the visitor is not logged in, then the permission cannot be checked; the 
visitor is redirected to the login page and then back to the page that requires 
permissions. 


9.3.2 Combining requirements 

Occasionally, it is necessary to combine requirements. This can be done via 
a generic requires decorator which takes a single argument, a true or false 
condition. For example, to give access to agents, but only on Tuesday: 
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@auth. requires(auth.has_membership(group_id=agents) \ 

and request . now. weekday ( )==1) 
def function_seven( ) : 

return 'Hello agent, it must be Tuesday!' 


or equivalently: 


@auth. requires(auth.has_membership( role=' Secret Agent') \ 

and request . now. weekday ( )==1) 
def function_seven( ) : 

return 'Hello agent, it must be Tuesday!' 


9.3.3 Authorization and CRUD 

Using decorators and /or explicit checks provides one way to implement 
access control. 

Another way to implement access control is to always use CRUD (as opposed 
to SQLFORM) to access the database and to ask CRUD to enforce access control 
on database tables and records. This is done by linking Auth and CRUD with 
the following statement: 

crud. settings .auth = auth 

This will prevent the visitor from accessing any of the CRUD functions unless 
the visitor is logged in and has explicit access. For example, to allow a visitor 
to post comments, but only update their own comments (assuming crud, 
auth and db. comment are defined): 


def 

give. 

_create_permission(foi 

-m) : 







group_id 

= auth.id_group( 

'user. 

_%s' % auth. 

user. id) 




auth 

.add. 

.permission(group_ 

-id, 

' read' , 

db. comment) 




auth 

.add. 

.permission(group. 

-id, 

' create 

, db 

.comment) 




auth 

.add. 

.permission(group. 

-id, 

'select 

, db 

.comment) 



def 

give. 

_update_permission(foi 

-m) : 







comment_id = form.vars.id 








group_id 

= auth.id_group( 

'user. 

_%s' % auth. 

user. id) 




auth 

.add. 

_permission(group_ 

-id, 

'update 

, db 

.comment, 

comment. 

-id) 


auth 

.add. 

.permission(group_ 

-id, 

'delete 

, db 

.comment, 

comment. 

-id) 

auth. settings . register_onaccept = 

give_create_ 

permission 


crui 

i. settings .auth = auth 
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15 
16 

def post_comment ( ) : 

*7 

form = crud . create {db. comment, onaccept=give_update_permission) 

18 

comments = db(db. comment) . select ( ) 

19 

return diet (form=form, comments=comments) 

20 
21 

def update_comment ( ) : 

22 

form = crud . update{db. comment, request .args(O) ) 

23 

return diet (form=form) 


You can also select specific records (those you have 'read' access to): 

1 def post_comment ( ) : 

2 form = crud . create (db. comment, onaccept=give_update_permission) 
1 query = auth.accessible_query( ' read' , db. comment, auth. user. id) 

4 comments = db(query) .select (db. comment .ALL) 

5 return diet (form=form, comments=comments) 


The permissions names enforced by : 

crud. settings .auth = auth 

are "read", "create", "update", "delete", "select", "impersonate". 

9.3.4. Authorization and downloads 

The use of decorators and the use of crud. settings. auth do not enforce 
authorization on files downloaded by the usual download function 

def download(): return response. download (request, db) 

If one wishes to do so, one must declare explicitly which "upload" fields 
contain files that need access control upon download. For example: 

db. def ine_t able ( 'dog' , 

Field ( 'small-image' , 'upload'), 
Field (' large^image' , 'upload')) 

db. dog. large_image. authorization = lambda record: \ 
auth.is_logged_in( ) and \ 
auth. has_permission( ' read' , db.dog, record. id, auth. user. id) 

The attribute authorization of upload field can be None (the default) or a 
function that decides whether the user is logged in and has permission 
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to 'read' the current record. In this example, there is no restriction on 
downloading images linked by the "small_image" field, but we require access 
control on images linked by the "large_image" field. 

9.3.5 Access Control and Basic Authentication 

Occasionally, it may be necessary to expose actions that have decorators that 
require access control as services; i.e., to call them from a program or script 
and still be able to use authentication to check for authorization. 

Auth enables login via basic authentication: 

] auth. settings .allow_basic_login = False 

With this set, an action like 

1 @auth. requires_login( ) 

2 def give_me_time( ) : 
i import time 

4 return time.ctime() 

can be called, for example, from a shell command: 

1 wget --user=[username] - -password=[password] 
z http://. . ./[app]/[controller]/give_me_time 

Basic login is often the only option for services (described in the next 
chapter), but it is disabled by default. 

9.3.6 Manual Authentication 

Some times you want to implement your own logic and do "manual" user 
login. This can also be done by calling the function: 

1 user = auth.login_bare(username : password) 

login_bare returns user if the user exists and the password is valid, else it 
returns None, username is the email if the "auth_user" table does not have a 
"username" field. 
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9.3.7 Settings and messages 

Here is a list of all parameters that can be customized for Auth 

The following must point to a gluon. tools. Mail object to allow auth to send 
emails: 

auth. settings .mailer = None 

The following must be the name of the controller that defined the user action: 
auth. settings . controller = 'default' 

The following is a very important setting: 
auth. settings . hmac_key = None 

It must be set to something like "sha5i2:a-pass-phrase" and it will be passed 
to the CRYPT validator for the "password" field of the auth_user table. It will 
be the algorithm and a-pass-phrase used to hash the passwords. 

By default, auth also requires a minimum password length of 4. This can be 
changed: 


auth 

.settings 

.password. 

min_length = 

4 


To disabled 

an action 

append its 

name 

to this list: 

auth 

.settings 

. actions_d 

isabled = [] 




For example: 

auth. settings .actions_disabled .append) ' register' ) 

will disable registration. 

If you want to receive an email to verify registration set this to True: 
auth. settings . registration_requires_verification = False 

To automatically login people after registration, even if they have not 
completed the email verification process, set the following to True: 

auth. settings .login_after_ registration = False 
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If new registrants must wait for approval before being able to login set this 
to True: 
] auth. settings . registration_requires_approval = False 

Approval consists of setting registration_key==" via appadmin or 
programmatically. 

If you do not want a new group for each new user set the following to False: 

1 auth. settings . create_user_groups = True 

The following settings determine alternative login methods and login forms, 
as discussed previously: 

1 auth. settings .login_methods = [auth] 

2 auth. settings .login_form = auth 

Do you want to allow basic login? 

1 auth. settings .allows_basic_login = False 

The following is the URL of the login action: 
1 auth. settings .login_url = URL( 'user', args=' login' ) 

If the user tried to access the register page but is already logged in, he will 
be redirected to this URL: 
1 auth. settings .loggecLurl = URL( 'user', args='profiZe' ) 

This must point to the URL of the download action, in case the profile 
contains images: 
1 auth. settings .downloacLurl = URL ( 'download' ) 

These must point to the URL you want to redirect your users to after the 
various possible auth actions (in case there is no referrer): 

1 auth. settings .login_next = URL( 'index' ) 

2 auth. settings .logout_next = URL( 'index') 

3 auth. settings . prof ile_next = URL( 'index') 

4 auth. settings . register_next = URL( 'user', args=' login ' ) 

5 auth. settings . retrieve_username_next = URL( 'index' ) 

6 auth. settings . retrieve_password_next = URL( 'index' ) 

7 auth. settings . change_password_next = URL( 'index' ) 
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8 auth. settings . request_reset_password_next = URL( 'user', a rgs=' login' ) 
,., auth. settings . reset_password_next = URL( 'user', args=' login' ) 
in auth. settings . verify_email_next = URL( 'user', args=' login' ) 

If the visitor is not logger in, and calls a function that requires 
authentication, the user is redirected to auth. settings. login_url which 
defaults to URL( 'default ', 'user/login' ). One can replace this behavior by 
redefining: 

■ auth. settings .on_failed_authentication = lambda url: redirect(url) 

This is the function called for the redirection. The argument url' passed to 
this function is the url for the login page. 

If the visitor does not have permission to access a given function, the visitor 
is redirect to the URL defined by 

■ auth. settings .on_failed_authorization = \ 

2 URL( 'user' , args='on_failecLaurhorization' ) 

You can change this variable and redirect the user elsewhere. 

Often on_failed_authorization is a URL but it can be a function that returns 
the URL and it will be called on failed authorization. 

These are lists of callbacks that should be executed after form validation for 
each of the corresponding action before any database IO: 

■ auth. settings .login_onvalidation = [] 

z auth. settings . register_onvalidation = [] 

3 auth. settings .prof ile_onvalidation = [] 

4 auth. settings . retrieve_password_onvalidation = [] 

5 auth. settings . reset_password_onvalidation = [] 

Each callback must be a function that takes the form object and it can modify 
the attributes of the form object before database IO is performed. 

These are lists of callbacks that should be executed after the database IO is 
performed and before redirection: 

1 auth. settings .login_onaccept = [] 

2 auth. settings . register_onaccept = [] 

3 auth. settings .prof ile_onaccept = [] 

4 auth. settings . verify_email_onaccept = [] 
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Here is an example: 

auth. settings . register_onaccept .append (lambda form:\ 
mail. send [to= l you@example. com' ,subject='new user' , 

message="new user email is %s'%form.vars. email)) 


You can enable captcha for any of the auth actions: 


auth. settings . captcha = None 


auth. settings .login_captcha = None 


auth. settings . register_captcha = None 


auth. settings . retrieve. use rname_captcha 

= None 

auth. settings . retrieve. pas sword_captcha 

= None 


If the .captcha settings points to a gluon. tools. Recaptcha, all forms for which 
the corresponding option (like . login_captcha) is set to None will have a 
captcha, while those for which the corresponding option is set to False 
will not. If, instead, .captcha is set to None, only those form who have a 
corresponding option set to a gluon. tools. Recaptcha object will have captcha 
and the others will not. 

This is the login session expiration time: 

auth. settings .expiration = 3600 # seconds 

You can change the name of the password field (in Firebird for example 
"password" is a keyword and cannot be used to name a field): 

auth. settings . password_field = 'password' 

Normally the login form tries to validate an email. This can be disabled by 

changing this setting: 

auth. settings .login_email_ validate = True 

Do you want to show the record id in the edit profile page? 
auth. settings . showid = False 

For custom forms you may want to disable automatic error notification in 
forms: 

auth. settings . hideerror = False 

Also for custom forms you can change the style: 
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] auth. settings . formstyle = ' table3cols' 

(it can be "table2cols", "divs" and "ul") 

And you can set the separator for auth-generated forms: 
1 auth. settings .label_separator = ' : ' 

By default the login form gives the option to extend the login via "remember 
me" option. The expiration time can be changed or the option disabled via 
these settings: 

1 auth. settings .long_expiration = 3600*24*30 # one month 

2 auth. settings . remember_me_form = True 

You can also customize the following messages whose use and context should 
be obvious: 

1 auth. messages . submit_button = 'Submit' 

2 auth. messages . verify_password = 'Verify Password' 

3 auth. messages .delete. label = 'Check to delete:' 

4 auth. messages . function_disabled = 'Function disabled' 

5 auth. messages .access_denied = 'Insufficient privileges' 

6 auth. messages . registration_verifying = 'Registration needs verification' 

7 auth. messages . registration_pending = 'Registration is pending approval' 
s auth. messages .login_disabled = 'Login disabled by administrator' 

,., auth. messages .logged_in = 'Logged in' 

w auth. messages .email_sent = 'Email sent' 

i] auth. messages .unable_to_send_email = 'Unable to send email' 

12 auth. messages .email_verified = 'Email verified' 

n auth. messages .logged_out = 'Logged out' 

14 auth. messages . registration_successful = 'Registration successful' 

15 auth. messages .invalid_email = 'Invalid email' 

10 auth. messages . unable_send_email = 'Unable to send email' 

17 auth. messages .invalid_login = 'Invalid login' 

is auth. messages .invalid_user = 'Invalid user' 

n, auth. messages .is_empty = "Cannot be empty" 

.0 auth. messages .mismatched_password = "Password fields don't match" 

2, auth. messages . verify_email = ... 

22 auth. messages . verify_email_subject = 'Password verify' 

23 auth. messages . username_sent = 'Your username was emailed to you' 

24 auth. messages . new_password_sent = 'A new password was emailed to you' 
.-, auth. messages . password_changed = 'Password changed' 

20 auth. messages . retrieve_username = 'Your username is: %(username)s' 

27 auth. messages . retrieve_username_subject = 'Username retrieve' 

■^ auth. messages . retrieve_password = 'Your password is: %(password)s' 

29 auth. messages . retrieve_password_subject = 'Password retrieve' 
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30 auth. messages . reset_password = ... 

31 auth. messages . reset_password_subject = 'Password reset' 

32 auth. messages .invalid_reset_password = 'Invalid reset password' 

33 auth. messages .profile_updated = 'Profile updated' 

34 auth. messages . new_password = 'New password' 

35 auth. messages .old_password = 'Old password' 

36 auth. messages. group_description = \ 

37 'Group uniquely assigned to user %(id)s' 

38 auth. messages . register. log = 'User %(id)s Registered' 

39 auth. messages .login_log = 'User %(id)s Logged-in' 

40 auth. messages .logout_log = 'User%(id)s Logged-out' 

4 1 auth. messages . profile. log = 'User %(id)s Profile updated' 

42 auth. messages . verify_email_log = 'User %(id)s Verification email sent' 

43 auth. messages . retrieve_username_log = 'User%(id)s Username retrieved' 

44 auth. messages . retrieve_password_log = 'User %(id)s Password retrieved' 

45 auth. messages . reset_password_log = 'User %(id)s Password reset' 

4 b auth. messages . change_password_log = 'User %(id)s Password changed' 

47 auth. messages .add_group_log = 'Group %(group-id)s created' 

48 auth. messages .del_group_log = 'Group %(group-id)s deleted' 

49 auth. messages .add_membership_log = None 

50 auth. messages .del_membership_log = None 

51 auth. messages . has_membership_log = None 
v auth. messages .add_permission_log = None 

53 auth. messages .del_permission_log = None 

54 auth. messages . has_permission_log = None 

55 auth. messages .label_first_name = 'First name' 

56 auth. messages .label_last_name = 'Last name' 

57 auth. messages .label_username = 'Username' 

58 auth. messages .label_email = 'E-mail' 

59 auth. messages .label_password = 'Password' 

60 auth. messages . label. registration_key = 'Registration key' 

61 auth. messages .label_reset_password_key = 'Reset Password key' 

62 auth. messages . label- registration_id = 'Registration identifier' 

63 auth. messages .label_role = 'Role' 

(, 4 auth. messages .label_description = 'Description' 

65 auth. messages .label_user_id = 'User ID' 

66 auth. messages .label_group_id = 'Group ID' 

67 auth. messages .label_name = 'Name' 

68 auth. messages .label_table_name = 'Table name' 

69 auth. messages .label_record_id = 'Record ID' 

70 auth. messages .label_time_stamp = 'Timestamp' 

71 auth. messages .label_client_ip = 'Client IP' 

72 auth. messages .label_origin = 'Origin' 

73 auth. messages .label_remember_me = "Remember me (for 30 days)" 


add | del | has membership logs allow the use of "%(user_id)s" and 
"%(group_id)s". add | del | has permission logs allow the use of "%(user_id)s", 
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"%(name)s", "%(table_name)s", and "%(record_id)s" 


9.4 Central Authentication Service 

web2py provides support for third party authentication and single sign 
on. Here we discuss the Central Authentication Service (CAS) which is an 
industry standard and both client and server are built-into web2py 

CAS is an open protocol for distributed authentication and it works in the 
following way: When a visitor arrives at our web site, our application 
check in the session if the user is already authenticated (for example via a 
session. token object). If the user is not authenticated, the controller redirects 
the visitor from the CAS appliance, where the user can log in, register, and 
manage his credentials (name, email and password). If the user registers, he 
receives an email, and registration is not complete until he responds to the 
email. Once the user has successfully registered and logged in, the CAS 
appliance redirects the user to our application together with a key. Our 
application uses the key to get the credentials of the user via an HTTP request 
in the background to the CAS server. 

Using this mechanism, multiple applications can use a single sign-on via a 
single CAS server. The server providing authentication is called a service 
provider. Applications seeking to authenticate visitors are called service 
consumers. 

CAS is similar to OpenID, with one main difference. In the case of OpenID, 
the visitor chooses the service provider. In the case of CAS, our application 
makes this choice, making CAS more secure. 

Running a web2py CAS provider is as easy as copying the scaffolding app. 
In fact any web2py app that exposes the action 

# in provider app 

def user(): return dict(form=auth( ) ) 

is a CAS 2.0 provider and its services can be accessed at the URL 

http://. . ./provider/default/user/cas/login 
http://. . ./provider/default/user/cas/validate 
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http://. . ./provider/default/user/cas/logout 


(we assume the app to be called "provider"). 

You can access this service from any other web application (the consumer) 
by simply delegating authentication to the provider: 

1 # in consumer app 

z auth = Auth(db : cas_provider = ' http://127 .0.8. 1-.80BQ/ provider/ default/ user/ cas' ) 

When you visit the login url the consumer app, it will redirect you to the 
provider app which will perform authentication and will redirect back to the 
consumer. All processes of registration, logout, change password, retrieve 
password, have to be completed on the provider app. An entry about the 
logged-in user will be created on the consumer side so that you add extra 
fields and have a local profile. Thanks to CAS 2.0 all fields that are readable 
on the provider and have a corresponding field in the auth_user table of the 
consumer will be copied automatically. 

Auth( . . . ,cas_provider=' . . . ' ) works with third party providers and supports 
CAS 1.0 and 2.0. The version is detected automatically. By default it 
builds the URLs of the provider from a base (the cas_provider url above) 
by appending 

] /login 
z /validate 
3 /logout 

These can be changed in consumer and in provider 



If you want to connect to a web2py CAS provider from a different domain, 
you must enable them by attending to the list of allowed domain: 


# in provider app 

auth. settings . cas_domains .append! 'example, com' ) 
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9.4.1 Using webzpy to authorize non-webzpy apps 

This is possible but dependent on the web server, here we assume two 
applications running under the same web server: Apache with mocLwsgi. One 
of the applications is web2py with an app proving access control via Auth. 
The other can be a CGI script, a PHP program or anything else. We want to 
instruct the web server to ask permission to the former application when a 
client requests access to the latter. 

First of all we need to modify the web2py application and add the following 
controller: 

1 def check_access() : 

z return 'true' if auth. is_logged_in( ) else 'false' 

which returns true if the user is logged in and false otherwise. Now run a 
web2py process in background: 

] nohup python web2py.py -a ' ' -p 8002 

Port 8002 is a must and there is no need to enable admin so no admin 
password. 

Then we need to edit the Apache config file (for example 
"/etc/apache2/sites-available/default") and instruct apache so that when 
the non-web2py program is called, it should call the above check action 
instead and only if it returns true it should proceed and respond to the 
request, else if should deny access. 

Because web2py and the non-web2py application run under the same 
domain, if the user is logged into the web2py app, the web2py session cookie 
will be passed to Apache even when the other app is requested and will 
allow credential verification. 

In order to achieve this we need a script, "web2py /scripts/ access. wsgi" that 
can play this trick. The script ships with web2py All we need to do it tell 
apache to call this script, the URL of the application needing access control, 
and the location of the script: 

1 <VirtualHost *:80> 

2 WSGIDaemonProcess web2py user=www-data group=www-data 
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, WSGIProcessGroup web2py 

4 WSGIScriptAlias / /home/www-data/web2py/wsgihandler.py 

5 

6 AliasMatch "myapp/path/needing/authentication/myfile /path/to/myfile 

r <Directory /path/to/> 

s WSGIAccessScript /path/to/web2py/scripts/access.wsgi 

9 </Directory> 

10 </VirtualHost> 

Here "rhyapp/path/needing/authentication/rnyfile" is the regular 
expression that should match the incoming request and "/path/to/" is 
the absolute location of the web2py folder. 

The "access. wsgi" script contains the following line: 

■ URL_CHECK_ACCESS = ' http: / 1121 .0 .0 .l:8QQ2/%(app)s/ 'default/ 'check-access' 

which points to the web2py application we have requested but you can edit 
it to point to a specific application, running on a port other than 8002. 

You can also change the check_access() action and make its logic more 
complex. This action can retrieve the URL that was originally requested 
using the environment variable 
] request .env. request_uri 

and you can implement more complex rules: 

1 def check_access( ) : 

2 if not auth.is_logged_in( ) : 

3 return 'false' 

4 elif not user_has_access( request .env. request_uri) : 

5 return 'false' 

6 else: 

7 return 'true' 


10 

Services 


The W3C defines a web service as "a software system designed to support 
interoperable machine-to-machine interaction over a network". This is a 
broad definition, and it encompasses a large number of protocols designed 
not for machine-to-human communication, but for machine-to-machine 
communication such as XML, JSON, RSS, etc. 

In this chapter we discuss how to expose web services using web2py. If 
you are interested in examples of consuming third party services (Twitter, 
Dropbox, etc.) you should look into Chapter 9 and Chapter 14. web2py 
provides, out of the box, support for many protocols, including XML, JSON, 
RSS, CSV, XMLRPC, JSONRPC, AMFRPC, and SOAP, webapy can also be 
extended to support additional protocols. 

Each of those protocols is supported in multiple ways, and we make a 
distinction between: 

• Rendering the output of a function in a given format (for example XML, 
JSON, RSS, CSV) 

• Remote Procedure Calls (for example XMLRPC, JSONRPC, AMFRPC) 
20.2 Rendering a dictionary 
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10.1.1 HTML, XML, and JSON 


Consider the following action: 


1 def count ( ) : 

z session. counter = (session. counter or 0) + 1 

3 return diet (counter=session. counter, now=request .now) 


This action returns a counter that is increased by one when a visitor reloads 
the page, and the timestamp of the current page request. 

Normally this page would be requested via: 

. http : //127 . 0.0 . 1 : 8000/app/def ault/count 

and rendered in HTML. Without writing one line of code, we can ask web2py 
to render this page using different protocols by adding an extension to the 
URL: 

. http : //127 . . . 1 : 8000/app/def ault/count . html 

2 http : //127 . . . 1 : 8000/app/def ault/count . xml 

3 http : //127 . 0.0 . 1 : 8000/app/def ault/count . j son 

The dictionary returned by the action will be rendered in HTML, XML and 
JSON, respectively. 

Here is the XML output: 

] <document> 

z <counter>3</counter> 

3 <now>2009-08-01 13:00:00</now> 

4 </document> 

Here is the JSON output: 

i { ' counter' :3, ' now' : ' 2009-08-01 13:60:00' } 

Notice that date, time, and datetime objects are rendered as strings in 
ISO format. This is not part of the JSON standard, but rather a web2py 
convention. 
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10.1.2 Generic views 

When, for example, the ".xml" extension is called, web2py looks for a 
template file called "default/count. xml", and if it does not find it, looks 
for a template called "generic. xml". The files "generic.html", "generic. xml", 
"generic.json" are provided with the current scaffolding application. Other 
extensions can be easily defined by the user. 

For security reasons the generic views are only allowed to be accessed on 
localhost. In order to enable the access from remote clients you may need to 
set the response. generic _patterns. 

Assuming you are using a copy of scaffold app edit the following line in 
models /db.py 

• restrict access only to localhost 


response. generic. patterns 

= [■*' 

] if 

request. 

is. 

Aocal 

else 

[] 

• to allow all generic views 

response. gene ric_patterns 

= ['* 

] 






• to allow onlyjson 

response. gene ric_patterns 

= ['*. 

json' 

] 






The generic_patterns is a glob pattern, it means you can use any patterns that 
matches with your app actions or pass a list of patterns. 
] response. generic_patterns = ['*. json' ,'*. xml' ] 

To use it in an older web2py app, you may need to copy the "generic.*" files 
from a later scaffolding app (after version 1.60). 

Here is the code for "generic.html" 

■ {{extend 'layout.html'}} 

2 

3 {{=BEAUTIFY(response._vars)}} 

4 

5 <button onclick=" document. location^' {{=URL(" admin", "default", "design", 

6 args=request . application) }} ' ">admin</button> 

7 <button onclick=" jQueryf '#request' ) ,slideToggle( ) ">request</button> 

s <div class="/iicfcfen" id="reguest"><h2>request</h2>{{=BEAUTIFY( request)}}</div> 
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<button onclick-" jQuery( '#session' ) .slideToggle( ) ">session</button> 

<div class=" hidden" id=" session "><h2>session</h2>{{=BEAUTI FY ( session )}}</div> 

<button onclick=" j Que ry ( '#response' ) . slideToggle( ) ">response</button> 

<div class=" hidden" id="response"><h2>response</h2>{{=BEAUTIFY( response) }}</div> 

<script>jQuery( ' .hidden ' ) .hide( ) ;</script> 


Here is the code for "generic. xml" 


1 

{{ 


2 

try: 


3 

from gluon. serializers import xml 


4 

response. write ( xml ( response._vars) 

• ,escape=False) 

5 

response. headers! 'Content-Type' ]= 

text/xml ' 

6 

except: 


7 
8 

raise HTTP ( 405, ' no xml') 
}} 



And here is the code for "generic.json" 



Any dictionary can be rendered in HTML, XML and JSON as long as it 
only contains python primitive types (int, float, string, list, tuple, dictionary), 
response. _vars contains the dictionary returned by the action. 

If the dictionary contains other user-defined or web2py-specific objects, they 
must be rendered by a custom view. 

10.1.3 Rendering Rows 

If you need to render a set of Rows as returned by a select in XML or JSON 
or another format, first transform the Rows object into a list of dictionaries 
using the as_list() method. 

Consider for example the following mode: 

db . def ine_table ( ' person ' , Field ( ' name ' ) ) 
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The following action can be rendered in HTML, but not in XML or JSON: 

1 def everybody ( ) : 

2 people = db( ). select(db. person. ALL) 

3 return diet (people=people) 

while the following action can rendered in XML and JSON: 

1 def everybody ( ) : 

2 people = db( ). select(db. person. ALL) .as_list( ) 

3 return diet (people=people) 


10.1.4 Custom formats 

If, for example, you want to render an action as a Python pickle: 

1 http : //127 . 0.0 . 1 : 8000/app/def ault/count . pickle 

you just need to create a new view file "default/countpickle" that contains: 

. {{ 

2 import cPickle 

3 response. headers! 'Content-Type' ] = ' application/ python. pickle' 

4 response. writ efcPickle. dumps ( response. _vars) : escape=False) 

> }} 

If you want to be able to render any action as a pickled file, you need only to 
save the above file with the name "genericpickle". 

Not all objects are pickleable, and not all pickled objects can be un-pickled. It 
is safe to stick to primitive Python objects and combinations of them. Objects 
that do not contain references to file streams or database connections are 
usually pickleable, but they can only be un-pickled in an environment where 
the classes of all pickled objects are already defined. 


10.1.5 RSS 

web2py includes a "generic. rss" view that can render the dictionary returned 
by the action as an RSS feed. 
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Because the RSS feeds have a fixed structure (title, link, description, items, 
etc.) then for this to work, the dictionary returned by the action must have 
the proper structure: 

1 {'title' 

2 'link' 

3 'description' 

4 ' created-on' 
j 'entries' 

and each entry in entries must have the same similar structure: 

■ {'title' : ", 

2 ' link' : ' ' , 

3 'description' : 

4 ' created-on' : ' ' } 



For example the following action can be rendered as an RSS feed: 

1 

def feed() : 

2 

return diet (title="my feed", 

3 

\ink=" http :// feed . example . com" , 

4 

description="my first feed", 

5 

entries=[ 

6 

dict(title="my feed", 

7 

link=" http :// feed . example. com" , 

8 

description="my first feed") 

9 

]) 


by simply visiting the URL: 
1 http : //127 . . . 1 : 8000/app/def ault/f eed . rss 

Alternatively, assuming the following model: 

1 db. def ine_table (' rss_entry' , 

Field ('title'), 

3 FieUC link'), 

4 Field ( ' created-on ' , ' datetime ' ) , 

5 Field ( 'description ') ) 

the following action can also be rendered as an RSS feed: 

■ def feed(): 

2 return diet (title="my feed", 

t \ink=" http :// feed . example . com" , 

4 description="my first feed", 

5 entries=db( ) . select (db. rss_entry.ALL) . as_list() ) 
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The as_list() method of a Rows object converts the rows into a list of 
dictionaries. 

If additional dictionary items are found with key names not explicitly listed 
here, they are ignored. 

Here is the "generic. rss" view provided by web2py: 



As one more example of an RSS application, we consider an RSS aggregator 
that collects data from the "slashdot" feed and returns a new web2py rss feed. 

def aggregator() : 

import gluon.contrib.feedparser as feedparser 
d = feedparser. parse( 

" http :// rss . slashdot . org/ Slashdot /slashdot /to") 
return diet (title=d. channel .title, 
link = d. channel. link, 
description = d .channel. description, 
created_on = request. now, 
entries = [ 

dict(title = entry. title, 

link = entry. link, 

description = entry. description, 

created_on = request. now) for entry in d. entries]) 

It can be accessed at: 
http://127.0.0.1:8000/app/default/aggregator. rss 


10.1.6 CSV 

The Comma Separated Values (CSV) format is a protocol to represent tabular 
data. 


Consider the following model: 
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db.define_model( 'animal ' , 
Field ( 'species ' ) , 
Field ( 'genus ' ) , 
Field C family' )) 

and the following action: 

def animals ( ) : 

animals = db( ) .select (db. animal. ALL) 
return diet (animals=animals) 

web2py does not provide a "generic. csv"; you must define a custom view 
"default/animals. csv" that serializes the animals into CSV. Here is a possible 
implementation : 



Notice that one could also define a "generic. csv" file, but one would have to 
specify the name of the object to be serialized ("animals" in the example). 
This is why we do not provide a "generic. csv" file. 

20.2 Remote procedure calls 

web2py provides a mechanism to turn any function into a web service. 
The mechanism described here differs from the mechanism described before 
because: 

• The function may take arguments 

• The function may be defined in a model or a module instead of controller 

• You may want to specify in detail which RPC method should be supported 

• It enforces a more strict URL naming convention 

• It is smarter than the previous methods because it works for a fixed set of 
protocols. For the same reason it is not as easily extensible. 
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To use this feature: 

First, you must import and instantiate a service object. 

i from gluon. tools import Service 

2 service = Service!) 

This is already done in the "db.py" model file in the scaffolding application. 

Second, you must expose the service handler in the controller: 

■ def call() : 

z session. forget ( ) 

3 return service!) 

This is already done in the " default. py" controller of the scaffolding application. 
Remove session . forget ( ) if you plan to use session cookies with the services. 

Third, you must decorate those functions you want to expose as a service. 
Here is a list of currently supported decorators: 

1 @service. run 

2 (aservice. xml 

3 (aservice. json 

4 (aservice. rss 

5 (aservice. csv 

6 @service.xmlrpc 

7 @service. jsonrpc 

s @service.amf rpc3( 'domain' ) 

,., @service. soap( ' FunctionName' , returns={ ' result' :type},args={ ' pa rami ' :type,}) 

As an example, consider the following decorated function: 

i (aservice. run 

2 def concat (a, b) : 

3 return a+b 

This function can be defined in a model or in the controller where the call 

action is defined. This function can now be called remotely in two ways: 

i http ://127.0.0.1:8000/app/default/caU/run/concat?a=hello&b=wo rid 
2 http : //127 . 0.0 . 1 : 8000/app/def ault/caU/run/concat/hello/world 

In both cases the http request returns: 

i helloworld 

If the (aservice. xml decorator is used, the function can be called via: 
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. http ://127. 0.0.1: 8000/app/def ault/call/xml/concat?a=hello&b=wo rid 
2 http : //127 . 0.0 . 1 : 8000/app/def ault/call/xml/concat/hello/world 

and the output is returned as XML: 

1 <document> 

2 <result>helloworld</result> 

3 </document> 

It can serialize the output of the function even if this is a DAL Rows object. 
In this case, in fact, it will call as_list( ) automatically. 

If the @service . j son decorator is used, the function can be called via: 

■ http -J/121 . 0.0.1: 8000/app/def ault/caU/json/concat?a=heUo&b=wo rid 
z http ://127. 0.0.1: 8000/app/def ault/call/json/concat/hello/wo rid 

and the output returned as JSON. 

If the @service.csv decorator is used, the service handler requires, as the 
return value, an iterable object of iterable objects, such as a list of lists. Here 
is an example: 

1 (aservice. csv 

2 def tablel(a.b) : 

3 return [ [a, b] , [1,2] ] 

This service can be called by visiting one of the following URLs: 

i http : //127 . . . 1 : 8000/app/def ault/call/csv/tablel?a=hello&b=wo rid 
2 http : //127 . . . 1 : 8000/app/def ault/call/csv/tablel/hello/wo rid 

and it returns: 

1 hello, world 

2 1,2 

The @service. rss decorator expects a return value in the same format as the 
"generic. rss" view discussed in the previous section. 

Multiple decorators are allowed for each function. 

So far, everything discussed in this section is simply an alternative to the 
method described in the previous section. The real power of the service 
object comes with XMLRPC, JSONRPC and AMFRPC, as discussed below. 
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10.2.1 XMLRPC 

Consider the following code, for example, in the "default. py" controller: 

i @service.xmlrpc 

2 def add(a,b) : 

3 return a+b 

4 

5 @service.xmlrpc 

6 def div(a,b) : 

7 return a/b 


Now in a python shell you can do 

i >» from xmlrpclib import Serve rProxy 

2 >» server = ServerProxy( 

3 'http://127.Q.O.l:8006/app/default/call/xmlrpc') 

4 »> print server. add(3, 4) 
s 7 

6 >» print server. add( 'hello' , 'world' ) 

7 ' helloworld' 

8 >» print server. div(12, 4) 

9 3 

io >» print server. div(l,0) 

i] ZeroDivisionError: integer division or modulo by zero 

The Python xmlrpclib module provides a client for the XMLRPC protocol. 
web2py acts as the server. 

The client connects to the server via ServerProxy and can remotely call 
decorated functions in the server. The data (a,b) is passed to the function(s), 
not via GET/POST variables, but properly encoded in the request body using 
the XMLPRC protocol, and thus it carries with itself type information (int or 
string or other). The same is true for the return value(s). Moreover, any 
exception raised on the server propagates back to the client. 

There are XMLRPC libraries for many programming languages (including C, 
C++, Java, C#, Ruby, and Perl), and they can interoperate with each other. 
This is one the best methods to create applications that talk to each other 
independent of the programming language. 

The XMLRPC client can also be implemented inside a web2py action, so that 
one action can talk to another web2py application (even within the same 
installation) using XMLRPC. Beware of session deadlocks in this case. If an 
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action calls via XMLRPC a function in the same app, the caller must release 
the session lock before the call: 
session . forget ( response) 


10.2.2 JSONRPC 

In this section we are going to use the same code example as for XMLRPC 
but we will expose the service using JSONRPC instead: 

@service. jsonrpc 
def add(a,b) : 
return a+b 

def call() : 

return service!) 

JSONRPC is very similar to XMLRPC but uses JSON instead of XML as data 
serialization protocol. 

Of course we can call the service from any program in any language 
but here we will do it in Python. web2py ships with a module 
"gluon/contrib/simplejsonrpc.py" created by Mariano Reingart. Here is an 
example of how to use to call the above service: 

>» from gluon.cont rib. simplej son rpc import 

>» URL = "http://127.0.0.1:8000/app/default/call/jsonrpc" 

>» service = ServerProxy(URL : verbose=True) 

>» print service. add(l, 2) 


10.2.3 JSONRPC and Pyjamas 

JSONRPC is very similar to XMLRPC, but uses the JSON-based protocol 
instead of XML to encode the data. As an example of application here, we 
discuss its usage with Pyjamas. Pyjamas is a Python port of the Google 
Web Toolkit (originally written in Java). Pyjamas allows writing a client 
application in Python. Pyjamas translates this code into JavaScript. web2py 
serves the JavaScript and communicates with it via AJAX requests originating 
from the client and triggered by user actions. 
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Here we describe how to make Pyjamas work with web2py. It does not 
require any additional libraries other than web2py and Pyjamas. 

We are going to build a simple "todo" application with a Pyjamas client (all 
JavaScript) that talks to the server exclusively via JSONRPC. 

First, create a new application called "todo". 

Second, in "models/db.py", enter the following code: 

1 db=DAL( ' sqlite: //storage. sqlite' ) 

2 db. define_table(' todo' , Field (' task' ) ) 

3 service = ServiceO 

(Note: Service class is from gluon.tools). 

Third, in "controllers/default. py", enter the following code: 

i def index( ) : 

redirect (URL (' todoApp 1 )) 

3 

4 @service. jsonrpc 

5 def getTasks( ) : 

6 todos = db(db.todo) .select( ) 

7 return [ (todo. task, todo. id) for todo in todos] 

8 

i) @service. jsonrpc 

io def addTask(taskFromJson) : 
u db. todo.insert(task= taskFromJson) 
12 return getTasks() 

1.3 

i 4 @service. jsonrpc 

i 5 def deleteTask (idFromJson) : 

16 del db. todo[idFromJson] 

17 return getTasks() 

18 

io def call() : 

20 session. f o rget ( ) 

21 return service)) 

22 

23 def todoApp( ) : 

24 return dict( ) 

The purpose of each function should be obvious. 

Fourth, in "views/default/todoApphtml", enter the following code: 

i <html> 
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<head> 

<meta name="pygwt: module" 

content^" {{=URL( ' static 1 , ' output/TodoApp' )}}" /> 
<title> 

simple todo application 
</title> 
</head> 

<body bgcolor='V/?ite"> 
<hl> 

simple todo application 
</hl> 
<i> 

type a new task to insert in db, 
click on existing task to delete it 
</i> 
<script language="_/'ai/ascript" 

src="{{=URL( 'static' , ' output/ pygwt.js' )}}"> 
</script> 
</body> 
</html> 

This view just executes the Pyjamas code in "static/output/todoapp" - code 
that we have not yet created. 

Fifth, in "static/TodoApp.py" (notice it is TodoApp, not todoApp!), enter the 
following client code: 

from pyjamas . ui.RootPanel import RootPanel 

from pyjamas . ui. Label import Label 

from pyjamas .ui.VerticalPanel import VerticalPanel 

from pyjamas . ui.TextBox import TextBox 

import pyjamas .ui.KeyboardListener 

from pyjamas . ui. ListBox import ListBox 

from pyjamas.ui.HTML import HTML 

from pyjamas .JSONService import JSONProxy 

class TodoApp: 

def onModuleLoad(self ) : 

self. remote = DataService( ) 
panel = VerticalPanel ( ) 

self . todoTextBox = TextBox () 

self . todoText Box. addKey boa rdListener( self ) 

self .todoList = ListBoxO 
self . todoList . setVisibleltemCount (7) 
self . todoList . setWidth ( "286px" ) 
self .todoList . addClickListener( self ) 
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self. Status = Label("") 

23 

24 panel .add (Label ( "Add New Todo:")) 

25 panel .add(self.todoTextBox) 

26 panel .add (Label ("Click to Remove:")) 

27 panel . add ( self .todoList) 

28 panel .add(self. Status) 

29 self . remote. getTasks( self ) 
3° 

31 RootPanel( ) .add (panel) 
32 

33 def onKeyLlp(self , sender, keyCode, modifiers): 

34 pass 
35 

3 6 def onKeyDown(self , sender, keyCode, modifiers): 

37 pass 
38 

3 9 def onKeyPress(self , sender, keyCode, modifiers): 

4° 

41 This function handles the onKeyPress event, and will add the 

42 item in the text box to the list when the user presses the 

43 enter key. In the future, this method will also handle the 

44 auto complete feature. 
45 

4 f, if keyCode = KeyboardListener.KEY_ENTER and \ 

47 sender == self . todoTextBox: 

48 id = self . remote. addTask(sender. getText( ), self ) 

49 sender. setText ( " " ) 

50 if id<0: 

51 RootPanel( ) . add(HTML( "Server Error or Invalid Response")) 
52 

53 def onClickfself , sender): 

54 id = self . remote. deleteTask( 

55 sender. get Value (sender. get Selectedlndex( ) ) ,self ) 

56 if id<0: 

57 RootPanel( ) .add( 

58 HTML( "Server Error or Invalid Response")) 
59 

60 def onRemoteResponse(self , response, request_info) : 

61 self .todoList .clear( ) 

62 for task in response: 

63 self. todoList. addltem( task [0]) 

64 self .todoList . setValue(self .todoList .get ItemCount ( ) -1, 

65 task[l]) 

66 

67 def onRemoteError(self , code, message, request_info) : 

68 self .Status. setText ( "Server Error or Invalid Response: " \ 

69 + "ERROR " + code + " - " + message) 
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7 1 

class DataService(JSONProxy) : 


7 2 

def __init__(self) : 


73 

JSONProxy.__init (self, "../. 

./ default/ call/ j son rpc" , 

74 

["getTasks" 

, "addTask" ."deleteTask"]) 

75 
76 

if name == ' main ': 


77 

app = TodoApp( ) 


78 

app.onModuleLoad( ) 



Sixth, run Pyjamas before serving the application: 

cd /path/to/todo/static/ 

python /python/pyjamas-0.5pl/bin/pyjsbuild TodoApp.py 

This will translate the Python code into JavaScript so that it can be executed 
in the browser. 

To access this application, visit the URL: 

http : //127 . . . 1 : 8000/todo/def ault/todoApp 

This subsection was created by Chris Prinos with help from Luke Kenneth 
Casson Leighton (creators of Pyjamas), updated by Alexei Vinidiktov. It has 
been tested with Pyjamas o.^pi. The example was inspired by this Django 
page in ref. [77]. 


10.2.4 Amfrpc 

AMFRPC is the Remote Procedure Call protocol used by Flash clients to 
communicate with a server. web2py supports AMFRPC, but it requires that 
you run web2py from source and that you preinstall the PyAMF library. This 
can be installed from the Linux or Windows shell by typing: 
easy_install pyamf 

(please consult the PyAMF documentation for more details). 

In this subsection we assume that you are already familiar with ActionScript 
programming. 

We will create a simple service that takes two numerical values, adds 
them together, and returns the sum. We will call our web2py application 
"pyamf_test", and we will call the service addNumbers. 
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First, using Adobe Flash (any version starting from MX 2004), create the Flash 
client application by starting with a new Flash FLA file. In the first frame of 
the file, add these lines: 


1 import mx. remoting. Service; 

2 import mx. rpc.RelayResponder; 

3 import mx. rpc. FaultEvent; 

4 import mx. rpc.ResultEvent ; 

5 import mx. remoting. PendingCall; 

6 

- var vail = 23; 

8 var val2 = 86; 

9 

10 service = new Service) 

" http:// 127. B.B.I: 8000/ pyamf Ltest /default/ call/ amf rpc3" , 
12 null, "mydomain" , null, null); 

13 

14 var pc: PendingCall = service. addNumbers(vall, val2); 

[ 5 pc.responder = new RelayResponder(this, "onResult" , "onFault") ; 

16 

1- function onResult ( re:ResultEvent) : Void { 

18 trace( "Result : " + re. result); 

19 txt_result . text = re. result; 

20 } 

21 

22 function onFault (fault :FaultEvent) :Void { 

23 trace( "Fault: " + fault .fault .faultstring) ; 

^4 } 

26 stopO; 


This code allows the Flash client to connect to a service that corresponds to 
a function called "addNumbers" in the file "/pyamf_test/default/gateway". 
You must also import ActionScript version 2 MX remoting classes to enable 
Remoting in Flash. Add the path to these classes to the classpath settings in 
the Adobe Flash IDE, or just place the "mx" folder next to the newly created 
file. 

Notice the arguments of the Service constructor. The first argument is 
the URL corresponding to the service that we want will create. The third 
argument is the domain of the service. We choose to call this domain 
"mydomain". 
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Second, create a dynamic text field called "txt_result" and place it on the 
stage. 

Third, you need to set up a web2py gateway that can communicate with the 
Flash client defined above. 

Proceed by creating a new web2py app called pyamf_test that will host the 

new service and the AMF gateway for the flash client. Edit the "default.py" 

controller and make sure it contains 

(aservice.amf rpc3( ' mydomain' ) 
def addNumbers(vall, val2) : 
return vail + val2 

def call(): return service() 

Fourth, compile and export /publish the SWF flash client as pyamf_test . swf, 
place the "pyamf_test.amf", "pyamf_test.html", "AC_RunActiveContent.js", 
and "crossdomain.xml" files in the "static" folder of the newly created 
appliance that is hosting the gateway, "pyamf_test". 

You can now test the client by visiting: 

http : //127 . 0.0 . 1 : 8000/pyamf_test/static/pyamf_test . html 

The gateway is called in the background when the client connects to 
addNumbers. 

If you are using AMFo instead of AMF3 you can also use the decorator: 

(aservice.amf rpc 

instead of: 

(aservice.amf rpc3( 'mydomain' ) 

In this case you also need to change the service URL to: 
http : //127 . 0.0 . 1 : 8000/pyamf_test/def ault/call/amf rpc 


10.2.5 SOAP 

web2py includes a SOAP client and server created by Mariano Reingart. It 
can be used very much like XML-RPC: 
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Consider the following code, for example, in the "default.py" controller: 

1 (aservice. soap( 'My Add' , returns={ ' result' :int} : args={ 'a ' :int , ' b' :int, }) 

2 def add(a,b) : 

3 return a+b 

Now in a python shell you can do: 

, >» from gluon .contrib.pysimplesoap. client import SoapClient 

2 >» client = SoapClient(wsdl="/)trp://locaZhost:S000/app/default/c3Zl/soap?WSDL") 

3 >» print client .MyAdd(a=l,b=2) 

4 {'result' : 3} 

To get proper encoding when returning a text values, specify string as 
u'proper utf8 text'. 

You can obtain the WSDL for the service at 

■ http : //127 . 0.0 . 1 : 8000/app/def ault/call/soap?WSDL 

And you can obtain documentation for any of the exposed methods: 
i http : //127 . 0.0 . 1 : 8000/app/def ault/call/soap 


20.3 Low level API and other recipes 

10.3.1 simplejson 

web2py includes gluon. contrib. simplejson, developed by Bob Ippolito. This 
module provides the most standard Python-JSON encoder-decoder. 

Simple JSON consists of two functions: 

• gluon. contrib. simples j son. dumps (a) encodes a Python object a into JSON. 

• gluon. contrib. simple j son. loads (b) decodes a JavaScript object b into a 
Python object. 

Object types that can be serialized include primitive types, lists, and 
dictionaries. Compound objects can be serialized with the exception of user 
defined classes. 
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Here is a sample action (for example in controller "default.py") that serializes 
the Python list containing weekdays using this low level API: 

1 def weekdays () : 

z names= [ ' Sunday ' , ' Monday ' , ' Tuesday ' , ' Wednesday ' , 

3 'Thursday' , 'Friday' , 'Saturday' ] 

4 import gluon.contrib.simplejson 

5 return gluon.contrib.simplejson.dumps(names) 

Below is a sample HTML page that sends an Ajax request to the above action, 
receives the JSON message and stores the list in a corresponding JavaScript 
variable: 

■ {{extend ' layout. html ' }} 

2 <script> 

, $. get JSON ( ' /application/ default/ weekdays' , 

4 function(data){ alert(data); }); 

5 </script> 

The code uses the jQuery function $. get JSON, which performs the Ajax call 
and, on response, stores the weekdays names in a local JavaScript variable 
data and passes the variable to the callback function. In the example the 
callback function simply alerts the visitor that the data has been received. 


10.3.2 PyRTF 

Another common need of web sites is that of generating Word-readable 
text documents. The simplest way to do so is using the Rich Text Format 
(RTF) document format. This format was invented by Microsoft and it has 
since become a standard. web2py includes gluon.contrib.pyrtf, developed by 
Simon Cusack and revised by Grant Edwards. This module allows you to 
generate RTF documents programmatically including colored formatted text 
and pictures. 

In the following example we instantiate two basic RTF classes, Document and 
Section, append the latter to the former and insert some dummy text in the 
latter: 


def makertf ( ) : 

import gluon.contrib.pyrtf as q 
doc=q. Document ( ) 
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In the end the Document is serialized by q.dumps(doc). Notice that before 
returning an RTF document it is necessary to specify the content-type in the 
header else the browser does not know how to handle the file. 

Depending on the configuration, the browser may ask you whether to save 
this file or open it using a text editor. 

10.3.3 ReportLab and PDF 

web2py can also generate PDF documents, with an additional library called 
"ReportLab" [76]. 

If you are running web2py from source, it is sufficient to have ReportLab 
installed. If you are running the Windows binary distribution, you need to 
unzip ReportLab in the "web2py/" folder. If you are running the Mac binary 
distribution, you need to unzip ReportLab in the folder: 

web2py.app/Con tents/Resources/ 

From now on we assume ReportLab is installed and that web2py can find it. 
We will create a simple action called "get_me_a_pdf" that generates a PDF 
document. 


from 

reportlab. platypus import * 



from 

report lab . lib . styles 

import 

getSampleStyleSheet 

from 

reportlab. rl_config 

import 

defaultPageSize 


from 

reportlab . lib . units 

import 

inch, mm 


from 

report lab. lib. enums 

import 

TA_LEFT, TA_RIGHT, 

TA_CENTER, TA_JUSTIFY 

from 

reportlab. lib import 

colors 



from 

uuid import uuid4 




from 

cgi import escape 




import os 




def i 

get_me_a_pdf () : 
title = "This The Doc 

Title" 



1 

leading = "First Paragraph" 
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text = 'bla '* 10000 

styles = getSampleStyleSheet ( ) 

tmpfilename=os. path. join (request. folder, 'private' ,str(uuid4( ) ) ) 

doc = SimpleDocTemplate(tmpfilename) 

story = [] 

story. append (Paragraph (escape (title) , styles ['Title"] ) ) 

story. append (Paragraph (escape (heading) , styles ["Heading2" ] ) ) 

story. append (Paragraph (escape (text) , styles ["Normal"] ) ) 

story. append ( Space r(l,2*inch) ) 

doc.build(story) 

data = open(tmpfilename, " rb" ) . read( ) 

os.unlink(tmpfilename) 

response. headers! 'Content-Type' ]=' application/ pdf 

return data 

Notice how we generate the PDF into a unique temporary file, tmpfilename, 
we read the generated PDF from the file, then we deleted the file. 

For more information about the ReportLab API, refer to the ReportLab 
documentation. We strongly recommend using the Platypus API of 
ReportLab, such as Paragraph, Spacer, etc. 


20.4 Restful Web Services 

REST stands for "REpresentational State Transfer" and it is a type of web 
service architecture and not, like SOAP, a protocol. In fact there is no 
standard for REST. 

Loosely speaking REST says that a service can be thought of as a collection 
of resources. Each resource should be identified by a URL. There are four 
methods actions on a resource and they are POST (create), GET (read), PUT 
(update) and DELETE, from which the acronym CRUD (create-read-update- 
delete) stands for. A client communicates with the resource by making an 
HTTP request to the URL that identifies the resource and using the HTTP 
method POST /PUT /GET /DELETE to pass instructions to the resource. The 
URL may have an extension, for example json that specify how the protocol 
for encoding the data. 

So for example a POST request to 
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http : //127 .0.0. 1/myapp/def ault/api/person 

means that you want to create a new person. In this case a person may 
correspond to a record in table person but may also be some other type of 
resource (for example a file). 

Similarly a GET request to 

http : //127 .0.0. 1/myapp/def ault/api/persons . j son 

indicates a request for a list of persons (records from the data person) in json 
format. 

A GET request to 

http : //127 .0.0. 1/myapp/def ault/api/person/1 .json 

indicates a request for the information associated to person/1 (the record with 
id==l) and in json format. 

In the case of web2py each request can be split into three parts: 

• A first part that identify the location of the service, i.e. the action that 
exposes the service: 

http : //127 . . . 1/myapp/def ault/api/ 

• The name of the resource (person, persons, person/1, etc.) 

• The communication protocol specified y the extension. 

Notice that we can always use the router to eliminate any unwanted prefix 
in the URL and for example simplify this: 
http : //127 .0.0. 1/myapp/def ault/api/person/1 .json 

into this: 

http : //127 . 0.0 . 1/api/person/l . j son 

yet this is a matter of test and we have already discussed it at length in 
chapter 4. 

In our example we used an action called api but this is not a requirement. We 
can in fact name the action that exposes the RESTful service any way we like 
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and we can in fact even create more than one. For the same of the argument 
we will continue to assume that our RESTful action is called api. 

We will also assume we have defined the following two tables: 

1 db.define_table( 'person ' , Field ( 'name' ) , Field ( 'info' ) ) 

2 db.define_table( 'pet' , Field ( 'owner' , db. person) , Field ( 'name' ) , Field ( ' info' ) ) 

and they are the resources we want to expose. 

The first thing we do is create the RESTful action: 

1 def api( ) : 

2 return locals( ) 

Now we modify it so that the extension is filtered out of the request args 
(so that request. args can be used to identify the resource) and so that it can 
handle the different methods separately: 

1 @request. restful ( ) 

2 def api() : 

3 def GET(*args,**vars) : 

4 return dict( ) 

5 def POST(*args,**vars) : 

6 return dict() 

7 def PUT(*args,**vars) : 

8 return dict() 

i) def DEI_ETE(*args,**vars) : 

10 return dict() 

u return locals( ) 

Now when we make a GET http request to 
. http : //127 . 0.0 . 1 : 8000/myapp/def ault/api/person/1 . j son 

it calls and returns GET ( 'person' , '1' ) where GET is the function defined 
inside the action. Notice that: 

• we do not need to define all four methods, only those that we wish to 
expose. 

• the method function can take named arguments 

• the extension is stored in request .extension and the content type is set 
automatically. 
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The @request. restful () decorator makes sure that the extension in the path 
info is stored into request. extension, maps the request method into the 
corresponding function within the action (POST, GET, PUT, DELETE), and 
passes request, a rgs and request, vars to the selected function. 

Now we build a service to POST and GET individual records: 


@rei 

quest. restful( ) 




def 

api() : 

response, view = 'gene. 

def GET(tablename,id) 

ric.json' 




if not tablename= 

=' person' 

: raise 

HTTP (400) 


return diet (person = 

db. person 

(id)) 



def POST(tablename,** 

fields): 




if not tablename= 

=' person' 

: raise 

HTTP (400) 


return db. person. 

validate.. 

and_ insert (**fields) 


return locals() 





Notice that: 

• the GET and POST are dealt with by different functions 

• the function expect the correct arguments (un-named arguments parsed 
by request, a rgs and named arguments are from request .vars) 

• they check the input is correct and eventually raise an exception 

• GET perform a select and returns the record, db. person (id). The output is 
automatically converted to JSON because the generic view is called. 

• POST performs a validate_and_insert( . . ) and returns the id of the new 
record or, alternatively, validation errors. The POST variables, **fields, 
are the post variables. 


10.4.1 parse_as_rest (experimental) 

The logic explained so far is sufficient to create any type of RESTful web 
service yet web2py helps us even more. 

In fact, web2py provides a syntax to describe which database tables we want 
to expose and how to map resource into URLs and vice versa. 

This is done using URL patterns. A pattern is a string that maps the request 
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args from a URL into a database query. There 4 types of atomic patterns: 

• String constants for example "friend" 

• String constant corresponding to a table. For example "friend [person]" 
will match "friends" in the URL to the "person" table. 

• Variables to be used to filter. For example "(person. id}" will apply a 
db. person. name=={ person .id} filter. 

• Names of fields, represented by ":field" 

Atomic patters can be combined into complex URL patters using "/" such as 
in 

"/ friend [person ]/ {person. id}/ : field" 

which gives a url of the form 

http://. . . ./f riend/1/name 

Into a query for a person. id that returns the name of the person. Here 
"friend [person]" matches "friend" and filters the table "person", "{person. id}" 
matches "1" and filters "person. id==i". ":field" matches "name" and returns: 

db(db.person.id==l) .select ( ) . first ( ) .name 

Multiple URL patters can be combined into a list so that one single RESTful 
action can serve different types of requests. 

The DAL has a method parse_as_rest(pattern,args, vars) that given a list 
of patterns, the request. args and the request. vars matches the pattern and 
returns a response (GET only). 

So here is a more complex example: 
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u "/ f riend/ {person . name}/ pet [ pet . owner] / {pet . name}" , 

12 "/ friend/ {person . name} /pet [pet . owner] /{pet .name}/ : field" 

n 1 

i 4 parser = db.parse_as_rest (patterns, args.vars) 

i 5 if parser, status == 200: 

if, return diet (content=parser. response) 

i 7 else: 

is raise HTTP(parser. status, parser, error) 

i 9 def POST(table_name,**vars) : 

jo if table_name == 'person': 

21 return db. person. validate_and_insert(**vars) 

22 elif table_name == 'pet': 

23 return db.pet . validate_and_insert (**vars) 

24 else: 

25 raise HTTP (400) 

26 return locals ( ) 

Which understands the following URLs that correspond to the listed patters: 

• GET all persons 

, http://. . ./api/f riends 

• GET one person with name starting with "t" 
. http://. . ./api/friend/t 

• GET the "info" field value of the first person with name equal to "Tim" 

i http://. . ./api/f riend/Tim/info 

• GET a list of pets of the person (friend) above 
i http ://... /api/f riend/Tim/pets 

• GET the pet with name "Snoopy of person with name "Tim" 
, http://. . ./api/f riend/Tim/pet/Snoopy 

• GET the "info" field value for the pet 

, http://. . ./api/f riend/Tim/pet/Snoopy/info 


The action also exposes two POST urls: 

• POST a new friend 

• POST a new pet 
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If you have the "curl" utility installed you can try: 

1 $ curl -d "name=Tim" http://127.0.0. l:8000/myapp/default/api/f riend. json 

2 {"errors" : {}, "id": 1} 

3 $ curl http://127.0.0.1:8000/myapp/default/api/f riends. json 

4 {"content" : [{"info": null, "name": "Tim", "id": 1}]} 

5 $ curl -d "name=Snoopy&owner=l" http://127.0.0. l:8000/myapp/default/api/pet . json 

6 {"errors": {}, "id": 1} 

7 $ curl http://127.0.0. l:8000/myapp/default/api/f riend/Tim/pet/Snoopy. json 


8 {"content": [{"info": null, "owner": 1, "name": "Snoopy", "id": 1}]} 

It is possible to declare more complex queries such where a value in the URL 
is used to build a query not involving equality. For example 

patterns = [ 'f riends/{person . name, contains} ' maps 
, http:// /friends/i 

into 

i db. person. name. contains( 'i ' ) 


And similarly: 

patterns = [ 'f riends/{person . name. ge}/{person . name. gt. not} ' maps 
http://. . . ./f riends/aa/uu 

into 

(db. person. name>= ' aa ' )&(-(db. person. name>'uu' ) ) 

valid attributes for a field in a pattern are: contains, startswith, le, ge, It, gt, 
eq (equal, default), ne (not equal). Other attributes specifically for date and 
datetime fields are day, month, year, hour, minute, second. 

Notice that this pattern syntax is not designed to be general. Not every 
possible query can be described via a pattern but a lot of them are. The 
syntax may be extended in the future. 

Often you want to expose some RESTful URLs but you want to restrict the 
possible queries. This can be done by passing an extra argument queries to 
the parse_as_rest method, queries is a dictionary of (tablename, query) where 
query is a DAL query to restrict access to table tablename. 

We can also order results using the order GET variables 


services 455 


] http://. . . ./api/f riends?order=name|~info 

which order alphabetically (name) and then by reversed info order. 

We can also limit the number of records by specifying a limit and offset GET 
variables 

, http:// /api/f riends?offset=10&limit=1000 

which will return up to 1000 friends (persons) and skip the first 10. limit 
defaults to 1000 and offset default to o. 

Let's now consider an extreme case. We want to build all possible patterns 
for all tables (except auth_ tables). We want to be able to search by any text 
field, any integer field, any double field (by range) and any date (also by 
range). We also want to be able to POST into any table: 

In the general case this requires a lot of patterns. Web2py makes it simple: 

] (arequest. restful ( ) 

z def api( ) : 

3 response. view = 'generic. '+request. extension 

4 def GET(*args,**vars) : 

5 patterns = 'auto' 

6 parser = db.parse_as_rest (patterns, args.vars) 

7 if parser. status == 200: 

s return dict(content=parser. response) 

i) else: 

io raise HTTP(parser. status, parser, error) 

u def POST(table_name,**vars) : 

12 return db[table_name] . validate_and_insert (**vars) 

13 return locals () 

Settings patterns='auto' results in web2py generating all possible patterns 
for all non-auth tables. There is even a pattern for querying about patterns: 
i http://. . . ./api/patterns. json 

which for out person and pet tables results in: 
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"/ person/ id/ {person . id}/ pet [ pet . owner] / id/ {pet . id}/ : field" , 
"I person/ id/ {person . id} / pet [ pet . owner] / owner/ {pet . owner}" , 
"/ person/ id/ {person . id} / pet [ pet . owner] / owner/ {pet . owner}/ : field" 
"I person/ name/ pet [pet . owner] " , 
"/ person/ name/ pet [pet . owner ]/ id/ {pet . id} " , 
"/ person/ name/ pet [pet . owner] /id/ {pet. id}/ : field" , 
"/ person/ name/ pet [pet . owner ]/ owner/ {pet . owner}" , 
"/ person/ name/ pet [pet . owner ]/ owner/ {pet . owner}/ : field" , 
"/ person/ info/ pet [pet . owner] " , 
"/ person/ info/ pet [pet . owner] /id/ {pet. id} " , 
"/ person/ info/ pet [pet . owner] /id/ {pet. id} / : field" , 
"/ person/ info/ pet [pet . owner ]/ owner/ {pet . owner}" , 
"/ person/ info/ pet [pet . owner ]/ owner/ {pet . owner}/ : field" , 
"/pet [pet]" , 
" /pet/ id/ '{pet. id}" , 
"/pet/ id/ {pet. id}/ -.field" , 
"/ pet/ owner / {pet . owner}" , 
" /pet /owner /{pet .owner}/ : field" 
]} 

You can specify auto patterns for some tables only: 

patterns = [' :auto[person] ' ,' :auto[pet] ' ] 


20.4.2 smart_query (experimental) 

There are time when you need more flexibility and you want to be able to 
pass to a RESTful service an arbitrary query like 

http://. . ./api. json?search=person.name starts with 'T and person. name contains 'ra' 


You can do this using 
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The method db.smart_query takes two arguments: 

• a list of field or table that should be allowed in the query 

• a string containing the query expressed in natural language 

and it returns a db . set object with the records that have been found. 

Notice that the search string is parsed, not evaluated or executed and 
therefore it provides no security risk. 

10.4.3 Access Control 

Access to the API can be restricted as usual by using decorators. So, for 
example 

auth. settings .allow_basic_login = True 



can now be accessed with 

$ curl --user name: password http://127.0.0.1:8000/myapp/default/api/hello 
access granted, you said hello 


20.5 Services and Authentication 

In the previous chapter we have discussed the use of the following 
decorators: 

@auth. requires_login( ) 

@auth. requires_membership( . . . ) 

@auth. requires. pe mission ( . . . ) 

For normal actions (not decorated as services), these decorators can be used 
even if the output is rendered in a format other than HTML. 
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For functions defined as services and decorated using the ©service... 
decorators, the @auth. . . decorators should not be used. The two types of 
decorators cannot be mixed. If authentication is to be performed, it is the 
call actions that needs to be decorated: 

1 @auth. requires_login( ) 

z def calA(): return serviceO 

Notice that it also possible to instantiate multiple service objects, register 
the same different functions with them, and expose some of them with 
authentication and some not: 

1 public_services=Service( ) 

2 private_services=Service( ) 

3 

4 @public_service. jsonrpc 

5 @private_service. jsonrpc 
i, def f(): return 'public' 
7 

s @private_service. jsonrpc 
9 def g(): return 'private' 

u def public_call( ) : return public_service( ) 

[ 3 @auth. requires_login( ) 

i 4 def private_call( ) : return private_service( ) 

This assumes that the caller is passing credentials in the HTTP header (a valid 
session cookie or using basic authentication, as discussed in the previous 
section). The client must support it; not all clients do. 


11 

jQuery and Ajax 


While web2py is mainly for server-side development, the welcome 
scaffolding app comes with the base jQuery library [32], jQuery calendars 
(date picker, datetime picker and clock), the "superfish.js" menu, and some 
additional JavaScript functions based on jQuery. 

Nothing in web2py prevents you from using other Ajax [78] libraries such as 
Prototype, ExtJS, or YUI, but we decided to package jQuery because we find 
it to be easier to use and more powerful than other equivalent libraries. We 
also find that it captures the web2py spirit of being functional and concise. 


11.1 web2py_ajax.html 

The scaffolding web2py application "welcome" includes a file called 

views/web2py_a j ax . html 

which looks like this: 


{{ 


response. files .insert(0,URL( 'static' . 

, 'js/jquery.js')) 

response. files .insert(l,URL( 'static' . 

, ' css/calenadar. ess ' ) ) 

response. files .insert(2,URL( 'static' . 

, ' js/ calendar. js ' ) ) 

response. include_meta( ) 


response. include_files( ) 


}} 
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8 <script type="text/javascript"><\ - - 

i) // These variables are used by the web2py_ajax_init 

10 // function in web2py.js (which is loaded below). 

u var w2p_ajax_confirm_message = 

12 "{{=T( 'Are you sure you want to delete this object?' )}}" ; 

13 var w2p_ajax_date_format = "{{=T( '%Y-%m-%d' )}}" ; 

14 var w2p_ajax_datetime_format = "{{=T( '%Y-%m-%d %H:%M:%S' )}}" ; 
[ 5 //- -></script> 

iB <script src="{{=URL( ' static' ,' js/web2py.js' )}}" 
17 type="text/_/ai/ascript"x/script> 

This file is included in the HEAD of the default "layout.html" and it provides 
the following services: 

Includes "static/jquery.js". 

Includes "static/calendar.js" and "static/calendar.css", which are used for 
the popup calendar. 

Includes all response. meta headers 

Includes all response. files (requires CSS and JS, as declared in the code) 

Sets form variables and includes "static/js/web2y.js" 

web2py.js" does the following: 

Defines an ajax function (based on jQuery $.ajax). 

Makes any DIV of class "error" or any tag object of class "flash" slide down. 

Prevents typing invalid integers in INPUT fields of class "integer". 

Prevents typing invalid floats in INPUT fields of class "double". 

Connects INPUT fields of type "date" with a popup date picker. 

Connects INPUT fields of type "datetime" with a popup datetime picker. 

Connects INPUT fields of type "time" with a popup time picker. 

Defines web2py_a j ax_component, a very important tool that will be described 
in Chapter 12. 

Defines web2py_comet, a function that can be used for HTML5 websockets 
(not described in this book but read the examples in the source of 
"gluon/contrib/comet_messaging.py"). 
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It also includes popup, collapse, and fade functions for backward 
compatibility. 

Here is an an example of how the other effects play well together. 

Consider a test app with the following model: 

, db = DM.("sqlite://db.db") 

2 db.define_table( 'child' , 

1 Field ( ' name ' ) , 

4 Field ( 'weight' , 'double'), 

Field (' bi rth_date' , 'date'), 
ft Field (' time-Of-birth' , 'time')) 
7 

s db. child. name. requires=IS_NOT_EMPTY( ) 

9 db . child .weight . requires=IS_FLOAT_IN_RANGE (0,100) 

10 db. child. birth_date. requires=IS_DATE( ) 

11 db. child. time_of_ birth. requires=IS_TIME( ) 

with this "default.py" controller: 

1 def index( ) : 

form = SQLFORM(db. child) 
1 if form. process! ) .accepted: 

4 response. flash = ' record inserted' 

5 return diet (form=form) 

and the following "default /index. html" view: 

■ {{extend 'layout.html}} 
, {{-form}} 

The "index" action generates the following form: 


Name: 
Weigth: 
Birth Date: 
Time Of Birth: 


r 5ub.m it ^ 


If an invalid form is submitted, the server returns the page with a modified 
form containing error messages. The error messages are DIVs of class "error", 
and because of the above web2py.js code, the errors appears with a slide- 
down effect: 
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Name: 



enter a value 


Weigth: 

_| 



enter a number between 0.0 and 100.0 


Birth Date: 




enter date and time as 1963-08-28 1 4:30:59 


Time Of Birth: ' 

1 



enter time as hh:mm:ss (seconds, am, pm optional) 



[Submit J 


The color of the errors is given in the CSS code in "layout.html". 

The web2py.js code prevents you from typing an invalid value in the input 
field. This is done before and in addition to, not as a substitute for, the 
server-side validation. 

The web2pyjs code displays a date picker when you enter an INPUT field of 
class "date", and it displays a datetime picker when you enter an INPUT field 
of class "datetime". Here is an example: 


test 

I •* | ► | | + |#top://lZ7.0.0.1:3000/test/defaulT/lndex fi (Q- Google 

) 


test 

customize me! 

Index Edit 




Name: 



Weigth: 


Birth Date: 
Time Of Birth: 

? 




^— 

EE 


* 

-. 


. 


wk 

St-r Mdh Tue Wed Thu Fri Sa 


iz 

1 2 3 1 4 B 6 7 


31 
33 

33 

e 9 10 11 iz 13 11 

15 15 17 IB 19 ZD 21 
22 23 24 25 2& 27 2S 




Copyright O 2010 - Powered J 


3-1 

2S 30 31 


1 





The web2pyjs code also displays the following time picker when you try to 
edit an INPUT field of class "time": 
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^n^ 


test 




* 1 ► + >>http7/lZ7.0,0,LSOOO/resi/default/lridex 


<& | ' Q." Google 




customize mo! 







Index Edit 








Name: 




' 


Weigth: 






Birth Date: 






Time Of Birth: 




1 


Q « 

* 





1 Ifl 






- 

■ 






1. > .<: 


Upon submission, the controller action sets the response flash to the message 
"record inserted". The default layout renders this message in a DIV with 
id="flash". The web2pyjs code is responsible for making this DIV appear 
and making it disappear when you click on it: 



These and other effects are accessible programmatically in the views and via 
helpers in controllers. 


11.2 j Query effects 


The basic effects described here do not require any additional files; 
everything you need is already included in web2py_ajax.html. 
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HTML /XHTML objects can be identified by their type (for example a DIV), 
their classes, or their id. For example: 

1 <div class="one" id="a">Hello</div> 

2 <div class="two" id="f>">World</div> 

They belong to class "one" and "two" respectively. They have ids equal to "a" 
and "b" respectively. 

In jQuery you can refer to the former with the following a CSS-like equivalent 
notations 



and to the latter with 

1 jQuery( ' . two' ) 

2 jQuery( '#/)' ) 

3 jQuery( 'DIV. two 1 ) 

4 jQuery( 'DIV #b') 

or you can refer to both with 
1 jQuery( 'DIV') 

Tag objects are associated to events, such as "onclick". jQuery allows linking 
these events to effects, for example "slideToggle": 

1 <div class="one" id="a" onc\ick=" jQuery ( '. two' ) . slideToggle( ) ">Hello</div> 

2 <div class="two" id="b">World</div> 

Now if you click on "Hello", "World" disappears. If you click again, "World" 
reappears. You can make a tag hidden by default by giving it a hidden class: 

1 <div class="one" id="a" onclick="_/Ci/ery( ' . two') . slideToggle( ) ">Hello</div> 

2 <div class="two hidden" id="to">World</div> 

You can also link actions to events outside the tag itself. The previous code 
can be rewritten as follows: 

1 <div class="one" id="a">Hello</div> 

2 <div class="two" id="b">World</div> 

3 <script> 

4 jQuery( ' . one' ) . click ( function) ){jQuery( ' . two' ) . slideToggle ( )}) ; 
r, </script> 


JQUERY AND AJAX 465 

Effects return the calling object, so they can be chained. 

When the click sets the callback function to be called on click. Similarly for 
change, keyup, keydown, mouseover, etc. 

A common situation is the need to execute some JavaScript code only after 
the entire document has been loaded. This is usually done by the onload 
attribute of BODY but jQuery provides an alternative way that does not 
require editing the layout: 

1 <div class="one" id="a">Hello</div> 

2 <div class="two" id="f>">World</div> 
, <script> 

4 j Query (document) . ready ( function ( ){ 

5 jQuery( ' . one' ) .click( function! ){jQuery( ' . two' ) . slideToggle( )}) ; 

6 }); 

7 </script> 

The body of the unnamed function is executed only when the document is 
ready, after it has been fully loaded. 

Here is a list of useful event names: 

Form events 

• onchange: Script to be run when the element changes 

• onsubmit: Script to be run when the form is submitted 

• onreset: Script to be run when the form is reset 

• onselect: Script to be run when the element is selected 

• onblur: Script to be run when the element loses focus 

• onfocus: Script to be run when the element gets focus 
Keyboard events 

• onkeydown: Script to be run when key is pressed 

• onkeypress: Script to be run when key is pressed and released 

• onkeyup: Script to be run when key is released 
Mouse events 
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• onclick: Script to be run on a mouse click 

• ondblclick: Script to be run on a mouse double-click 

• onmousedown: Script to be run when mouse button is pressed 

• onmousemove: Script to be run when mouse pointer moves 

• onmouseout: Script to be run when mouse pointer moves out of an 
element 

• onmouseover: Script to be run when mouse pointer moves over an element 

• onmouseup: Script to be run when mouse button is released 
Here is a list of useful effects defined by jQuery: 

Effects 

jQuery(...).attr(name): Returns the name of the attribute value 

jQuery(...).attr(name, value): Sets the attribute name to value 

jQuery(...).show(): Makes the object visible 

jQuery(...).hide(): Makes the object hidden 

jQuery(...).slideToggle(speed, callback): Makes the object slide up or down 

jQuery(...).slideUp(speed, callback): Makes the object slide up 

jQuery(...).slideDown(speed, callback): Makes the object slide down 

jQuery(...).fadeIn(speed, callback): Makes the object fade in 

jQuery(...).fadeOut(speed, callback): Makes the object fade out 

The speed argument is usually "slow", "fast" or omitted (the default). The 
callback is an optional function that is called when the effect is completed. 
jQuery effects can also easily be embedded in helpers, for example, in a view: 

{{=DIV( 'click me.' 1 , -onc\ick="jQuery(this) . fadeOutO ")}} 


jQuery is a very compact and concise Ajax library; therefore web2py does 
not need an additional abstraction layer on top of jQuery (except for the 
ajax function discussed below). The jQuery APIs are accessible and readily 
available in their native form when needed. 
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Consult the documentation for more information about these effects and 
other jQuery APIs. 

The jQuery library can also be extended using plugins and User Interface 
Widgets. This topic is not covered here; see ref. [80] for details. 


11.2.1 Conditional fields informs 

A typical application of jQuery effects is a form that changes its appearance 
based on the value of its fields. 

This is easy in web2py because the SQLFORM helper generates forms that 
are "CSS friendly". The form contains a table with rows. Each row contains 
a label, an input field, and an optional third column. The items have ids 
derived strictly from the name of the table and names of the fields. 

The convention is that every INPUT field has an id tablename_f ieldname and 
is contained in a row with id tablename_f ieldname row. 

As an example, create an input form that asks for a taxpayer's name and for 
the name of the taxpayer's spouse, but only if he/she is married. 

Create a test application with the following model: 

, db = DAL('sc/lite://cfb.df)') 

z db.define_table( ' taxpayer' , 

3 Field ( ' name ' ) , 

4 Field ( 'married' , 'boolean'), 

5 Field ( 'spouse-name' ) ) 

the following "default.py" controller: 

1 def index( ) : 

2 form = SQLFORM(db. taxpayer) 

3 if form. process! ) .accepted: 

4 response. flash = ' record inserted' 

5 return diet (form=form) 

and the following "default /index. html" view: 


{{extend 

layout 

html'}} 


{{=form}} 




<script> 




jQuery(do< 

rument) 

. ready(fi 

inctionf ){ 
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-, jQuery( ' #taxpayer„spouse_name row' ) .hide() ; 

ft jQuery( ' #taxpaye remarried' ) . change ( function ( ){ 

if (j Query! ' #taxpayer_married' ) .attr( 'checked' ) ) 

8 jQuery( ' #taxpayer^spouse^name row' ) . show( ) ; 

9 else jQuery( '#taxpayer_spouse_name row' ) .hide() ;}) 

.0 }); 

11 </script> 


The script in the view has the effect of hiding the row containing the spouse's 
name: 



When the taxpayer checks the "married" checkbox, the spouse's name field 
reappears: 


nnn 


+ SjMtp://127.0.0.1:aaoo/te5t/default/index <5 | <Q? Google 



Here "taxpayer_married" is the checkbox associated to the "boolean" field 


JQUERY AND AJAX 469 


"married" of table "taxpayer". "taxpayer_spouse_name row" is the row 

containing the input field for "spouse_name" of table "taxpayer". 


11.2.2 Confirmation on delete 

Another useful application is requiring confirmation when checking a 
"delete" checkbox such as the delete checkbox that appears in edit forms. 

Consider the above example and add the following controller action: 



and the corresponding view "default/edit.html" 


1 {{extend 'layout.html'}} 

2 {{=fonn}} 


The deletable=True argument in the SQLFORM constructor instructs web2py 
to display a "delete" checkbox in the edit form. It is False by default. 
web2py's "web2pyjs" includes the following code: 

1 j Query (document) . ready ( function ( ){ 

2 jQuery( ' input. delete' ) .attr( 'onclick' , 

3 ' if (this, checked) if(!confirm( 

4 "{{=T( 'Sure you want to delete this object? ' )}}") ) 

5 this. checked=false; ' ) ; 

6 }); 


By convention this checkbox has a class equal to "delete". The jQuery code 
above connects the onclick event of this checkbox with a confirmation dialog 
(standard in JavaScript) and unchecks the checkbox if the taxpayer does not 
confirm: 
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^ I ► 4- ■i>http//l?7.0.0.1:8000/tesl/de(ayltfindex/l tf Q.- Google 


http://127.U.0.1:S<] Clo- 
sure you want to delete this object? 


( Cancel ^ ( OK ^ 


Name: N'^sino 

Married: Q 

Spouse Name: Claudia 

Check to delete: 

f Submit \ 


3 


11.3 TTie ajax function 

In web2py.js, web2py defines a function called ajax which is based on, but 
should not be confused with, the jQuery function $.ajax. The latter is much 
more powerful than the former, and for its usage, we refer you to ref. [32] 
and ref. [79]. However, the former function is sufficient for many complex 
tasks, and is easier to use. 

The aj ax function is a JavaScript function that has the following syntax: 
ajax(url, [namel, name2, ...], target) 

It asynchronously calls the url (first argument), passes the values of the field 
inputs withthe name equal to one of the names in the list (second argument), 
then stores the response in the innerHTML of the tag with the id equal to 
target (the third argument). 

Here is an example of a default controller: 

def one( ) : 

return diet ( ) 

def echo( ) : 

return request. vars. name 

and the associated "default/one.html" view: 


JQUERY AND AJAX 47I 


1 {{extend ' layout. html ' }} 

2 <form> 

3 <input name="naroe" onkeyup="a_/axrec/70' , ['name'], 'target')" /> 

4 </form> 

5 <div id="target"x/div> 

When you type something in the INPUT field, as soon as you release a key 
(onkeyup), the ajax function is called, and the value of the name="name" field 
is passed to the action "echo", which sends the text back to the view. The a j ax 
function receives the response and displays the echo response in the "target" 
DIV. 


11.3.1 Eval target 

The third argument of the ajax function can be the string ":eval". This means 
that the string returned by server will not be embedded in the document but 
it will be evaluated instead. 


Here is an example of a default controller: 


def one( ) : 


return diet ( ) 


def echo() : 


return " j Query ( '#target' ) .html(%s); 

' % rep r( request .vars . name) 


and the associated "default/onehtml" view: 



This allows for more complex responses that can update multiple targets. 


11.3.2 Auto-completion 

Web2py contains a built-in autocomplete widget, described in the Forms 
chapter. Here we will build a simpler one from scratch. 
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Another application of the above ajax function is auto-completion. Here we 
wish to create an input field that expects a month name and, when the visitor 
types an incomplete name, performs auto-completion via an Ajax request. In 
response, an auto-completion drop-box appears below the input field. 

This can be achieved via the following default controller: 

def month_input ( ) : 
return diet ( ) 

def month_selector( ) : 

if not request .vars. month : return '' 

months = ['January' , 'February' , 'March', 'April', 'May', 

'June', 'July', 'August', 'September' ,'October', 
'November', 'December'] 
month_start = request .vars .month. capitalize! ) 
selected = [m for m in months if m.startswith(month_start) ] 
return DIV(*[DIV(k, 

-onc\ick="jQuery('#month').val('%s')" % k, 
_onmouseover=" this. style . backgroundColor=' ye I low' " , 
_onmouseout=" this. style. backgroundColor=' white' " 
) for k in selected] ) 

and the corresponding "default/month_inputhtml" view: 

{{extend ' layout.html' }} 

<style> 

#suggestions { position: relative; } 

.suggestions { background: white; border: solid lpx #55A6C8; } 

.suggestions DIV { padding: 2px 4px 2px 4px; } 

</style> 

<form> 
<input type="texr" id="month" name="month" style='Vidth: 250px" /><br /> 
<div style="position: absolute;" id=" suggestions" 
class= " suggestions"></div> 
</form> 
<script> 
jfjuery( "#month" ) . keyup( function ( ){ 

ajax( 'month_selector' , ['month'], 'suggestions')}); 
</script> 

The jQuery script in the view triggers the Ajax request each time the visitor 
types something in the "month" input field. The value of the input field is 
submitted with the Ajax request to the "month_selector" action. This action 
finds a list of month names that start with the submitted text (selected), 
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builds a list of DIVs (each one containing a suggested month name), and 
returns a string with the serialized DIVs. The view displays the response 
HTML in the "suggestions" DIV. The "month_selector" action generates both 
the suggestions and the JavaScript code embedded in the DIVs that must be 
executed when the visitor clicks on each suggestion. For example when the 
visitor types "M" the callback action returns: 

<div> 

<div onclick="j Query ( '#month' ) .val( 'March' ) " 

onmouseout=" this. style. backgroundColor=' white' " 
onmouseover=" this. style. backgroundColor=' yellow' ">March</div> 
<div onclick="_/Query( '#month' ) .val( 'May' ) " 

onmouseout=" this. style. backgroundColor=' white' " 
onmouseover=" this. style. backgroundColor=' yellow' ">May</div> 
</div> 


Here is the final effect: 


test 


hltp:// 127.0,0, l:800<Vte5t/def< <T] (0? Google 



If the months are stored in a database table such as: 
db . def ine_table ( ' month ' , Field ( ' name ' ) ) 


then simply replace the month_selector action with: 


def month_input ( ) : 
return diet ( ) 


def month_selector( ) : 

if not request. vars. month : 
return ' ' 
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pattern = request .vars .month. capitalize! ) + '%' 

selected = [row. name for row in db(db. month. name. like(pattern) ) .select () ] 

return ' ' . join( [DIV(k, 

_onclick="_/Query ('#month' ) . val('%s ') " % k, 
_onmouseove r=" this. style . backgroun6Color=' yellow' " , 
_onmouseout=" this. style. backgroun6Color=' white' " 
).xml() for k in selected]) 

jQuery provides an optional Auto-complete Plugin with additional 
functionalities, but that is not discussed here. 


11.3.3 Ajaxform submission 

Here we consider a page that allows the visitor to submit messages using 
Ajax without reloading the entire page. Using the LOAD helper, web2py 
provides a better mechanism for doing it than described here, which will be 
described in Chapter 12. Here we want to show you how to do it simply 
using jQuery. 

It contains a form "myform" and a "target" DIV. When the form is submitted, 
the server may accept it (and perform a database insert) or reject it (because 
it did not pass validation). The corresponding notification is returned with 
the Ajax response and displayed in the "target" DIV. 

Build a test application with the following model: 


1 

db 

= DAL( 

sqlite 

■J 16b 

,6b') 





2 

db. 

,define_ 

.table( 

'post 

', Field ( 

'your 

-message ' , 

'text' )) 

3 

db. 

, post . you r_mes sage . 

requires ■■ 

= IS_ 

NOT. 

.EMPTY) ) 



Notice that each post has a single field "your_message" that is required to be 
not-empty 

Edit the default . py controller and write two actions: 


def 

index( ) : 
return dict() 



def 

new_post( ) : 

form = SQLFORM(db.post) 




if form.accepts(request, 

formname; 

=None) : 


return DIU( "Message f. 

lostecf" ) 



elif form. errors: 
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return TABLE(*[TR(k, v) for k, v in form. errors. items( )] ) 


The first action does nothing other than return a view. 

The second action is the Ajax callback. It expects the form variables in 
request . vars, processes them and returns DIV( "Message posted" ) upon success 
or a TABLE of error messages upon failure. 

Now edit the "default/indexhtml" view: 
] {{extend 'layout.html'}} 

3 <div id="target"x/div> 
4 

5 <form iti="myform"> 

6 <input name="your_message" id="your_message" /> 

7 <input type="submit" /> 

8 </form> 

[o <script> 

i] jQuery( 'ttmyform' ) . submit(function( ) { 
aj ax ( '{{=URLC new.post'W , 

13 [' your-wessage' ] , 'target'); 

14 return false; 

15 }); 

16 </script> 

Notice how in this example the form is created manually using HTML, but it 
is processed by the SQLFORM in a different action than the one that displays 
the form. The SQLFORM object is never serialized in HTML. SQLFORM. accepts 
in this case does not take a session and sets formname=None, because we chose 
not to set the form name and a form key in the manual HTML form. 

The script at the bottom of the view connects the "myform" submit button to 
an inline function which submits the INPUT with id="your_message" using 
the web2py ajax function, and displays the answer inside the DIV with 
id="target". 


11.3.4. Voting and rating 

Another Ajax application is voting or rating items in a page. Here we 
consider an application that allows visitors to vote on posted images. The 
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application consists of a single page that displays the images sorted according 
to their vote. We will allow visitors to vote multiple times, although it 
is easy to change this behavior if visitors are authenticated, by keeping 
track of the individual votes in the database and associating them with the 
request . env . remote_add r of the voter. 

Here is a sample model: 

1 db = DAL( ' sqlite :// images . db' ) 

2 db.define_table( 'item' , 

3 Field (' image ' , 'upload'), 

4 Field (' votes' , 'integer', default=0)) 

Here is the default controller: 

1 def list_items( ) : 

2 items = db( ). select(db. item. ALL, orderby=db. item. votes) 

3 return diet (items=items) 
4 

5 def download ( ) : 

6 return response. download (request, db) 
7 

8 def vote( ) : 

i) item = db. item [request . vars .id] 

10 new_ votes = item, votes + 1 

u item. update„record(votes=new_votes) 

12 return str(new_votes) 

The download action is necessary to allow the list_items view to download 
images stored in the "uploads" folder. The votes action is used for the Ajax 
callback. 



Here is the "default/ list_items.html" view: 

' 

{{extend ' layout.html ' }} 

3 

<formxinput type=" hidden" id="id" name="id" value="" /></form> 

4 

{{for item in items:}} 

5 

<p> 

6 

<img src=" {{=URL( ' download' , args-item. image)}}" 

7 

width="200px" /> 

8 

<br /> 

9 

Votes=<span id="item^Y=item. id»" >{{=item. votes }}</span> 

10 

[<span onclick=" jQueryf '#id' ) . val( ' {{=item.id}} ' ) ; 

[i 

ajax( ' vote' , ['id'], ' item{ {=i tern. id}} '); ">vote up</span>] 

12 

</p> 

13 

{{pass}} 


JQUERY AND AJAX 477 


When the visitor clicks on "[vote up]" the JavaScript code stores the item. id in 
the hidden "id" INPUT field and submits this value to the server via an Ajax 
request. The server increases the votes counter for the corresponding record 
and returns the new vote count as a string. This value is then inserted in the 
target item{{=item.id}} SPAN. 

Ajax callbacks can be used to perform computations in the background, but we 
recommend using cron or a background process instead (discussed in chapter 
4), since the web server enforces a timeout on threads. If the computation takes 
too long, the web server kills it. Refer to your web server parameters to set the 
timeout value. 


12 

Components and plugins 


Components and plugins are relatively new features of web2py, and there is 
some disagreement between developers about what they are and what they 
should be. Most of the confusion stems from the different uses of these terms 
in other software projects and from the fact that developers are still working 
to finalize the specifications. 

However, plugin support is an important feature and we need to provide 
some definitions. These definitions are not meant to be final, just consistent 
with the programming patterns we want to discuss in this chapter. 

We will try to address two issues here: 

• How can we build modular applications that minimize server load and 
maximize code reuse? 

• How can we distribute pieces of code in a more or less plugin-and-play 
fashion? 

Components address the first issue; plugins address the second. 


22.2 Components 

A component is a functionally autonomous part of a web page. 
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A component may be composed of modules, controllers and views, but there 
is no strict requirement other than, when embedded in a web page, it must 
be localized within an html tag (for example a DIV, a SPAN, or an IFRAME) 
and it must perform its task independently of the rest of the page. We 
are specifically interested in components that are loaded in the page and 
communicate with the component controller function via Ajax. 

An example of a component is a "comments component" that is contained 
into a DIV and shows users' comments and a post-new-comment form. 
When the form is submitted, it is sent to the server via Ajax, the list is 
updated, and the comment is stored server-side in the database. The DIV 
content is refreshed without reloading the rest of the page. 

The web2py LOAD function makes this easy to do without explicit 
JavaScript/Ajax knowledge or programming. 

Our goal is to be able to develop web applications by assembling components 
into page layouts. 

Consider a simple web2py app "test" that extends the default scaffolding app 
with a custom model in file "models/db_comments.py": 



one action in "controllers/comments. py" 

1 @auth. requires_login( ) 

2 def post( ) : 

3 return diet (form=crud. create(db. comment) , 

4 comment s=db(db. comment) .select ( ) ) 


and the corresponding "views/comments/post.html" 

{{extend 'layout.html'}} 

{{for comment in comments:}} 

<div class="comment"> 

on {{^comment .posted_on}} {{=comment .posted_by.first_name}} 
says <span class="comment_toody">{{=comment . body}}</span> 

</div> 
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{{pass}} 
{{=form}} 


You can access it as usual at: 

http : //127 . . . 1 : 8000/test/comments/post 


[ * I ► j I + WMtp://127.0.0.1.800[>/myapp/commenl5/po5t Sj (Q? Go 



on 2010-08-04 10:25:18 Massimo says My first comment 
Your comment: — 


Copyright © 201 - Powered by web2py 


So far there is nothing special in this action, but we can turn it into a 
component by defining a new view with extension ".load" that does not 
extend the layout. 

Hence we create a "views/comments/post.load": 

i {{#extend 'layout.html' <- notice this is commented out!}} 
i {{for comment in comments:}} 

3 <div class="comment"> 

4 on {{^comment .postecLon}} {{=comment .posted_by.first_name}} 
-, says <span class="comment_toody">{{=comment .body}}</span> 

i, </div> 
- {{pass}} 
s {{=form}} 

We can access it at the URL 

i http : //127 . . . 1 : 8000/test/comments/post . load 


and it will look like this: 
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http://12?.0.0.1:80Q0/myapp/comment5/post.load 
|_^_[^J I + *~*Mtp://127.0.0.1.B000/myapp/comment5/posl.]oad <5 | 'Q? Google ) 

on 2010-08-04 10:25:1 8 Massimo says My first common! 


Your comment: 


This is a component that we can embed into any other page by simply doing 


{{=l_0AD( ' comments ' , 'post, load' ,ajax=True)}} 


For example in "controllers/default.py" we can edit 


1 def index( ) : 

2 return diet ( ) 


and in the corresponding view add the component: 


{{extend 'layout.html'}} 

<p>{{=' bla '*100}}</p> 

{{=L0AD( ' comments ' , 'post. load' ,ajax=True)}} 


Visiting the page 


http : //127 . . . 1 : 8000/test/def ault/index 


will show the normal content and the comments component: 
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[ * J ► I I + |O h1tp:/,aZ7.D.t).l:8000/myiipp/[jefaul1/in dex 6 \ ( 0? Google " 


myapp 

example of a modular app 


bla bla bla bla b!a bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla 
bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla 
bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bli 

on 2010-08-04 10:25:18 Massimo says My first comment 
on 201 0-08-04 1 0:32:38 Massimo says My second comment 


Your comment 



The {{=L0AD( ...)}} component is rendered as follows: 

i <script type="text/javascript"><\ -- 

2 web2py_component {"/test/comment/post, load" , " C2827 '18984176" ) 

3 //--x/scriptxdiv id="c282718984176">loadinq . . .</div> 

(the actual generated code depends on the options passed to the LOAD 
function). 

The web2py_component(url,id) function is defined in "web2py_ajax.html" and 
it performs all the magic: it calls the url via Ajax and embeds the response 
into the DIV with corresponding id; it traps every form submission into the 
DIV and submits those forms via Ajax. The Ajax target is always the DIV 
itself. 

The full signature of the LOAD helper is the following: 

1 L0AD(c=Wone, f=' index', args=[], vars={}, 

2 extension=None, target=None, 
1 ajax=False : ajax_trap=False, 

4 url=None : user_signature=False, 

5 content^' loading. ..' ,**attr) : 


Here: 


the first two arguments c and f are the controller and the function that we 
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want to call respectively. 

• args and vars are the arguments and variables that we want to pass to the 
function. The former is a list, the latter is a dictionary. 

• extension is an optional extension. Notice that the extension can also be 
passed as part of the function as in f='index.load'. 

• target is the id of the target DIV. If it is not specified a random target id is 
generated. 

• a j ax should be set to True if the DIV has to be filled via Ajax and to False if 
the DIV has to be filled before the current page is returned (thus avoiding 
the Ajax call). 

• ajax_trap=True means that any form submission in the DIV must be 
captured and submitted via Ajax, and the response must be rendered 
inside the DIV. ajax_trap=False indicates that forms must be submitted 
normally, thus reloading the entire page. ajax_trap is ignored and 
assumed to be True if ajax=True. 

• url, if specified, overrides the values of c, f, args, vars, and extension 
and loads the component at the url. It is used to load as components 
pages served by other applications (which my or may not be created with 
web2py). 

• use r_ signature defaults to False but, if you are logged in, it should be set 
to True. This will make sure the ajax callback is digitally signed. This is 
documented in chapter 4. 

• content is the content to be displayed while performing the ajax call. It can 
be a helper as in content=lMG( . . ). 

• optional **attr (attributes) can be passed to the contained DIV. 

If no . load view is specified, there is a generic . load that renders the dictionary 
returned by the action without layout. It works best if the dictionary contains 
a single item. 

If you LOAD a component having the . load extension and the corresponding 
controller function redirects to another action (for example a login form), the 
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. load extension propagates and the new url (the one to redirect too) is also 
loaded with a . load extension. 

*Please note:* Because Ajax post does not support multipart forms, i.e. file 
uploads, upload fields will not work with the LOAD component. You 
could be fooled into thinking it would work because upload fields will 
function normally if POST is done from the individual component's. load 
view. Instead, uploads are done with ajax-compatible 3rd-party widgets and 
web2py manual upload store commands. 


12.1.1 Client-Server component communications 

When the action of a component is called via Ajax, web2py passes two HTTP 
headers with the request: 

1 web2py-component-location 

2 web2py-component-element 

which can be accessed by the action via the variables: 

1 request .env. http_web2py_component_location 

2 request . env . http_web2py_component_element 

The latter is also accessible via: 
j request. cid 

The former contains the URL of the page that called the component action. 
The latter contains the id of the DIV that will contain the response. 

The component action can also store data in two special HTTP response 
headers that will be interpreted by the full page upon response. They are: 

1 web2py-component-flash 

2 web2py-component-command 

and they can be set via: 

1 response. headers! ' web2py- component - flash' ] = '....' 

2 response. headers! 'web2py- component -command' ]-'... ' 

or (if the action is called by a component) automatically via: 
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1 response. flash=' . 

2 response. js=' 


The former contains text that you want to be flashed upon response. The 
latter contains JavaScript code that you want to be executed upon response. 
It cannot contain newlines. 

As an example, let's define a contact form component in 
"controllers/contact/ask. py" that allows the user to ask a question. 
The component will email the question to the system administrator, flash a 
"thank you" message, and remove the component from the page: 



The first four lines define the form and accept it. The mail object used for 
sending is defined in the default scaffolding application. The last four lines 
implement all the component-specific logic by getting data from the HTTP 
request headers and setting the HTTP response headers. 

Now you can embed this contact form in any page via 

{{=L0AD( ' contact' , 'ask. load' ,ajax=True)}} 

Notice that we did not define a .load view for our ask component. We 
do not have to because it returns a single object (form) and therefore 
the "generic. load" will do just fine. Remember that generic views are a 
development tool. In production you should copy "views /generic. load" into 
"views/contact/ask. load". 

We can block access to a function called via Ajax by digitally signing the URL 
using the user_signature argument: 
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{{=L0AD( ' contact' , 'ask. load' : ajax=True,user_signature=True)}} 


which add a digital signature to the URL. The digital signature must then be 
validated using a decorator in the callback function: 


@auth. requires„signature( ) 
def ask() : . . . 


12.1.2 Trapped Ajax links 

Normally a link is not trapped, and by clicking in a link inside a component, 
the entire linked page is loaded. Sometimes you want the linked page to be 
loaded inside the component. This can be achieved using the A helper: 

{{=A( ' linked page' ,_href='http :/ /example, com' , cid= request . cid)}} 

If cid is specified, the linked page is loaded via Ajax. The cid is the id of 
the html element where to place the loaded page content. In this case we set 
it to request. cid, i.e. the id of the component that generates the link. The 
linked page can be and usually is an internal URL generated using the URL 
command. 


22.2 Plugins 

A plugin is any subset of the files of an application. 
and we really mean any: 

• A plugin is not a module, is not a model, it is not a controller, is not a 
view, yet it may contain modules, models, controllers and /or views. 

• A plugin does not need to be functionally autonomous and it may depend 
on other plugins or specific user code. 

• A plugin is not a plugins system and therefore has no concept of registration 
nor isolation, although we will give rules to try to achieve some isolation. 

• We are talking about a plugin for your app, not a plugin for web2py 
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So why is it called a plugin? Because it provides a mechanism for packing a 
subset of an app and unpacking it over another app (i.e. plug-in). Under this 
definition, any file in your app can be treated as a plugin. 

When the app is distributed, its plugins are packed and distributed with it. 

In practice, the admin provides an interface for packing and unpacking 
plugins separately from your app. Files and folder of your application that 
have names with the prefix plugiruname can be packed together into a file 
called: 

web2py. plugin. name .w2p and distributed together. 



^Ej 

<- C? ft ;© 12 7-0- 0.1:8000/admin7defajlt/de5igin7niyapp 

a\\ 




^^^^U^^^^^^^^^^^^^^jf " Y" | 

About V Errors V Versicmirg V Logout V Help )^f 

1 

• EDIT APPLICATION "MYAPP" 



views languages sialic modules plugins 




OModels© 


O Controllers ■ 



O Views© 



•Languages • 



O Static files a 



OModules© 



OPIugins* 



download plugins 



Blugln_-eommflnta 



nplC'jc plL^ii" file- i:-sc?- ' ir- No i'ili? :l csc-r up 03.:' 






The files that compose a plugin are not treated by web2py any differently 
than other files except that admin understands from their names that they 
are meant to be distributed together, and it displays them in a separate page: 
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5 plugin myapp/comfnents 
G fi Q 127.0. D.l:8000/admin/defajlt/plusin/myapp/comments ■£■ -\ 

• PLU G IN "CQMM ENTS1 IN APPLIC ATJON . "M YAPP". 

O Models 

lESD ^. plugln_comm-a-nts,py defr&E tables commonl 


O Controllers 




Edit „ 

^ plug In. 

_cornrnents.pv exposes 

O Views 




Ed. „ 

plLgln_cornm->niu'posi.l 

ibnl extarfls layoi 

Jfc. § 

p| Ua ln_c r. 

1 merits/post,! 

nd 

Static files 




Then^rano 

static II l« 



O Modules 




Thenar* no 

ntniMM 




Yet as a matter of fact, by the definition above, these plugins are more general 
than those recognized as such by admin. 

In practice we will only be concerned with two types of plugins: 

• Component Plugins. These are plugins that contain components as defined 
in the previous section. A component plugin can contain one or 
more components. We can think for example of a plugin„comments that 
contains the comments component proposed above. Another example 
could be plugin_tagging that contains a tagging component and a tag-cloud 
component that share some database tables also defined by the plugin. 

• Layout Plugins. These are plugins that contain a layout view and the static 
files required by such layout. When the plugin is applied it gives the app 
a new look and feel. 

By the above definitions, the components created in the previous section, for 
example "controllers/contact. py", are already plugins. We can move them 
from one app to another and use the components they define. Yet they are 
not recognized as such by admin because there is nothing that labels them 
as plugins. So there are two problems we need to solve: 
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• Name the plugin files using a convention, so that admin can recognize 
them as belonging to the same plugin 

• If the plugin has model files, establish a convention so that the objects it 
defines do not pollute the namespace and do not conflict with each other. 

Let's assume a plugin is called name. Here are the rules that should be 
followed: 

Rule 1: Plugin models and controllers should be called, respectively 

• models/plugin_ttflme. py 

• controllers/plugin_ttflme. py 

and plugin views, modules, static, and private files should be in folders 
called, respectively: 

• views/plugin_name/ 

• modules/plugiri-ttflffze/ 

• static/plugin_Mflme/ 

• private/plugin_Mfl?ne/ 

Rule 2: Plugin models can only define objects with names that start with 

• plugin^name 

• PluginNflme 


Rule 3: Plugin models can only define session variables with names that start 
with 

• session. plugin_Mflme 

• session. PluginNflme 

Rule 4: Plugins should include license and documentation. These should be 
placed in: 

• static/p lugin_ttflme/license . html 

• static/plugin_Mflme/about .html 
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Rule 5: The plugin can only rely on the existence of the global objects defined 
in scaffolding "db.py", i.e. 

• a database connection called db 

• an Auth instance called auth 

• a Crud instance called crud 

• a Service instance called service 

Some plugins may be more sophisticated and have a configuration parameter 
in case more than one db instance exists. 

Rule 6: If a plugin needs configuration parameters, these should be set via a 
PluginManager as described below. 

By following the above rules we can make sure that: 

• admin recognizes all the plugin^name files and folder as part of a single 
entity. 

• plugins do not interfere with each other. 

The rules above do not solve the problem of plugin versions and 
dependencies. That is beyond our scope. 

12.2.1 Component plugins 

Component plugins are plugins that define components. Components 
usually access the database and define with their own models. 

Here we turn the previous comments component into a comments_plugin by 
using the same code we wrote before, but following all of the previous rules. 

First, we create a model called "models/plugin_commentspy": 

1 db.define_table( ' plugiri-comments-comment ' , 

2 Field (' body ',' text ' , label= 'Your comment' ) , 

Field ( 'posted-.on' , 'datetime', default=request .now) , 

4 Field ( 'posted_by' , db.auth_user, default=auth.user_id) ) 

5 db.plugin_ comments- comment . posted- on. writ able=False 

6 db. plugin_ comments. comment . posted- on. readable=False 
- db. plugin_ comment s_comment . posted_by.writable=False 
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8 db.plugi.n_ comment s_comment . pos ted_by. readable=False 

10 def plugin_comments( ) : 

u return LOAD( ' plugin^cowwents' , 'post' ,ajax=T rue) 


(notice the last two lines define a function that will simplify the embedding 
of the plugin) 

Second, we define a "controllers/plugin_comments.py" 

1 @auth. requires_login( ) 

2 def post( ) : 

3 comment = db.plugin_comments_comment 

4 return diet (form=crud. create(comment) , 

5 comments=db (comment) .select () ) 


Third, we create a view called "views/plugin_comments/post.load" 



Now we can use admin to pack the plugin for distribution. Admin will save 
this plugin as: 

web2py. plugin . comments. w2p 

We can use the plugin in any view by simply installing the plugin via the 
edit page in admin and adding this to our own views 

{{=plugin_ comments! )}} 

Of course we can make the plugin more sophisticated by having components 
that take parameters and configuration options. The more complex the 
components, the more difficult it becomes to avoid name collisions. The 
Plugin Manager described below is designed to avoid this problem. 
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12.2.2 Plugin manager 

The PluginManager is a class defined in gluon. tools. Before we explain how it 
works inside, we will explain how to use it. 

Here we consider the previous comments_plugin and we make it better. We 
want to be able to customize: 

db.plugin_ comment s_comment . body .label 

without having to edit the plugin code itself. 
Here is how we can do it: 

First, rewrite the plugin "models/plugin_comments.py" in this way: 

db.define_table( ' plugin_comments_comment ' , 

Field ( 'body' , ' text' ,label=plugin_comments . comments. body_label) , 
Field ( 'posted_on' , ' datetime' , default=request .now) , 
Field ( 'posted^by' , db.auth_user, default=auth.user_id) ) 

6 def plugin_comments( ) 

7 from gluon. tools import PluginManager 

s plugins = PluginManager( 'comments ' , body_label=' Your comment') 

9 

10 comment = db.plugin_comments_comment 

u comment . label=plugins . comments . body_label 

12 comment . posted_on.writable=False 

13 comment . posted_on . readable=False 

14 comment . posted_by.writable=False 

15 comment . posted_by . readable=False 

16 return LOAD( ' plugin-comments' , 'post. load' ,ajax=True) 

Notice how all the code except the table definition is encapsulated in a single 
function. Also notice how the function creates an instance of a PluginManager. 

Now in any other model in your app, for example in "models/db.py", you 
can configure this plugin as follows: 

1 from gluon. tools import PluginManager 

2 plugins = PluginManager( ) 

3 plugins. comments. body_label = Ti'Post a comment') 

The plugins object is already instantiated in the default scaffolding app in 
"models/db.py" 


5 
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The PluginManager object is a thread-level singleton Storage object of Storage 
objects. That means you can instantiate as many as you like within the same 
application but (whether they have the same name or not) they act as if there 
were a single PluginManager instance. 

In particular each plugin file can make its own PluginManager object and 
register itself and its default parameters with it: 
1 plugins = PluginManager) 'name' , paraml=' value' , param2=' value' ) 

You can override these parameters elsewhere (for example in 
"models/db.py") with the code: 

1 plugins = PluginManager( ) 

2 plugins. name. pa rami = 'other value' 

You can configure multiple plugins in one place. 

1 plugins = PluginManager( ) 

2 plugins. name. pa rami = '...' 

3 plugins. name. param2 = '...' 

4 plugins. namel. param3 = '...' 

5 plugins. name2. param4 = '...' 

6 plugins. name3. pa ram5 = '...' 

When the plugin is defined, the PluginManager must take arguments: the 
plugin name and optional named arguments which are default parameters. 
However, when the plugins are configured, the PluginManager constructor 
must take no arguments. The configuration must precede the definition of the 
plugin (i.e. it must be in a model file that comes first alphabetically). 


12.2.3 Layout plugins 

Layout plugins are simpler than component plugins because usually they do 
not contain code, but only views and static files. Yet you should still follow 
good practice: 

First, create a folder called "static/plugin_layout_nfl?ne/" (where name is the 
name of your layout) and place all your static files there. 

Second, create a layout file called "views/plugin_layout_nfl?ne/layout.html" 
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that contains your layout and links the images, CSS and JavaScript files in 
"static/plugin_layout_Mflme/" 

Third, modify the "views/ layout.html" so that it simply reads: 

■ {{extend ' plugin-layout-name/layout.html'}} 
z {{include}} 

The benefit of this design is that users of this plugin can install 
multiple layouts and choose which one to apply simply by editing 
"views/layout.html". Moreover, "views/layouthtml" will not be packed by 
admin together with the plugin, so there is no risk that the plugin will 
override the user's code in the previously installed layout. 


22.3 plugin_wiki 

DISCLAIMER: pluginjwiki is still very much under development and 
therefore we do not promise backward compatibility to the same level as for 
webzpy core functions. 

plugin_wiki is a plugin on steroids. What we mean is that it defines multiple 
useful components and it may change the way you develop your applications: 

You can download it from 

http://web2py . com/examples/static/web2py. plugin.wiki.w2p 

The idea behind plugin_wiki is that most applications include pages that 
are semi-static. These are pages that do not include complex custom logic. 
They contain structured text (think of a help page), images, audio, video, 
crud forms, or a set of standard components (comments, tags, charts, maps), 
etc. These pages may be public, require login or have other authorization 
restrictions. These pages may be linked by a menu or only be reachable via 
wizard form. plugin_wiki provides an easy way to add pages that fit in this 
category to your regular web2py applications. 

In particular plugin_wiki provides: 

• A wiki-like interface that allows to add pages to your app and reference 
them by a slug. These pages (which we will refer to as wiki pages) have 
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versions and are stored in the database. 

• Public and private pages (require login). If a page requires login, it may 
also require that the user have a particular group membership. 

• Three levels: 1,2,3. At level 1, pages can only include text, images, 
audio and video. At level 2, pages can also include widgets (these are 
components as defined in the previous section that can be embedded in 
wiki pages). At level 3, pages can also include web2py template code. 

• A choice of editing pages with the markmin syntax or in HTML using a 
WYSIWYG editor. 

• A collection of widgets: implemented as components. They are self 
documenting and they can be embedded as regular components in normal 
web2py views or, using a simplified syntax, into wiki pages. 

• A set of special pages (meta-code, meta-menu, etc.) that can be used to 
customize the plugin (for example define code the plugin should run, 
customize the menu, etc.) 

The welcome app plus the plugin_wiki can be thought of as a development 
environment in itself that is suitable for building simple web applications such 
as a blog. 

From here on we will assume the plugin_wiki is applied to a copy of the 
welcome scaffolding app. 

The first thing you notice after installing the plugin is that it adds a new 
menu item called pages. 

Click on the pages menu item and you will be redirected to the plugin action: 


http : //127 . 0.0 . 1 : 8000/myapp/plugin_wiki/index 
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System Pages 

Create New page 


Slug: 

From Template: 


Current Pages 


Pages started wilh undescor 

must contain code and it 

must contain a descriptior 

It conatins the content of the header. 


The plugin index page lists the pages created using the plugin itself and 
allows you to create new ones by choosing a slug. Try creating a home page. 
You will be redirected to 


http : //127 .0.0.1: 8000/myapp/plugin_wiki/page/home 


Click on create page to edit its content. 


Page: home Title: Home 

Active: <& Public: g [Attachments] [Widget Builder] 



By default, the plugin is at level 3, which means you can insert widgets as well 
as code in pages. By default it uses the markmin syntax to describe the page 
content. 
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iz.^.i markmin syntax 

Here is a primer for the markmin syntax: 


markmin 


html 


# title 

## subtitle 

### subsubtitle 

**bold** 

"italic" 

http://... 

http://. . .png 

http://. . .mp3 

http://. . .mp4 

qr:http://... 

embed: http://. . . 


<hl>title</hl> 

<h2>subtitle</h2> 

<h3>subsubtitle</h3> 

<strong>bold</strong> 

<i>italic</i> 

<a href="http://. . .com">http: . . .</a> 

<img src="http://. . .png" /> 

<audio src="http:// . . .mp3"x/audio> 

<video src="http:// . . .mp4"x/video> 

<a href="http://. . . "ximg src="qr code"/x/a> 

<if rame src="http:// . . . "x/if rame> 


Notice that links, images, audio, and video files are embedded automatically. 
For more information on MARKMIN syntax, please refer to Chapter 5. 

If the page does not exist, you will be asked to create one. 

The edit page allows you to add attachments to pages (i.e. static files) 


Attachments 






[Cbss] 

Isho* all attache 










Nana. 





■*» f Choose 


"»" 

"*" 


s, 


"" 

*"" 

mentl glf 

2™""""':., 



and you can link to them as 

[[mylink name attachment : 3. png] ] 


or embed them with 

[[myimage attachment :3. png center 200px]] 
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The size (2QQpx) is optional, center is not optional but it may be replaced by 
left or right. 

You can embed blockquoted text with 

2 this is blockquoted 

3 

as well as tables 


2 I I x 

3 1 x 1 

4 X I I 

5 


and verbatim text 


verbatim text 


You can also prepend an optional : class to the final ---or the final ". For 
blockquoted text and tables it will be translated in the class of the tag, for 
example: 

test 
:abc 

renders as 

<blockquote class="abc">test</blockquote> 

For verbatim text the class can be used to embed content of different types. 

You can, for example, embed code with syntax highlighting by specifying the 
language with : code-language 

def index(): return 'hello world' 
:code_python 


You can embed widgets: 
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2 name: widget_name 

3 attributel: valuel 

4 attribute2: value2 
-, ' ' :widget 


From the edit page you can click on "widget builder" to insert widgets from 
a list, interactively: 



(for a list of widgets see the next subsection). 

You can also embed web2py template language code: 

{{for i in range(10) :}}<hl>{{=i}}</hl>{{pass}} 
: template 


12.3.2 Page permissions 

When editing a page you will find the following fields: 

• active (defaults to True). If a page is not active, it will not be accessible to 
visitors (even if public). 

• public (defaults to True). If a page is public, it can be accessed by visitors 
without logging in. 

• role (defaults to None). If a page has a role, the page can be accessed 
only by visitors who are logged in and are members of the group with the 
corresponding role. 
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12.3.3 Special pages 

meta-menu contains the menu. If this page does not exist, weD2py uses the 
response. menu defined in "models/menu.py". The content of the meta-menu 
page overrides the menu. The syntax is the following: 


1 

Item 1 Name http://linkl.com 

2 

Submenu Item 11 Name http://linkll.com 

3 

Submenu Item 12 Name http://linkl2.com 

4 

Submenu Item 13 Name http://linkl3.com 

5 

Item 2 Name http://linkl.com 

6 

Submenu Item 21 Name http://link21.com 

7 

Submenu Item 211 Name http://link211.com 

8 

Submenu Item 212 Name http://link212.com 

9 

Submenu Item 22 Name http://link22.com 


Submenu Item 23 Name http://link23.com 


where the indentation determines the submenu structure. Each item is 
composed of the text of the menu item followed by a link. A link can be 
page: slug. A link None does not link to any page. Extra spaces are ignored. 


Here is anotht 

it example: 

Home 

page: home 

Search Engines 

None 

Yahoo 

http: //yahoo. com 

Google 

http://google.com 

Bing 

http://bing. com 

Help 

page:help 


This renders as follows: 


Home Search Engines Help 


meta-code is another special page and it must contain web2py code. This is 
an extension of your models, and in fact you can put model code here. It is 
executed when "models /plugin_wiki.py" code is executed. 

You can define tables in meta-code. 

For example, you can create a simple table "friends" by placing this in 

meta-code: 

db.define_table( ' friend' , Field ( 'name' , requires=IS_NOT_EMPTY( ) ) ) 
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and you can create a friend-management interface by embedding in a page 
of your choice the following code: 

1 # List of friends 

i name: jqgrid 
A table: friend 
-, ' ' :widget 

6 

r # New friend 

s 

9 name: create 

[o table: friend 

i] ' ' :widget 


The page has two headers (starting with #): "List of friends" and "New 
friend". The page contains two widgets (under the corresponding headers): a 
jqgrid widget that lists friends and a crud create widget to add a new friend. 



meta-header, meta-footer, meta- sidebar are not used by the default layout in 
"welcome/views/layout.html". If you want to use them, edit "layout.html" 
using admin (or the shell) and place the following tags in the appropriate 
places: 


{{=plugin_wiki . embed_page ( 

'meta-header' ) 

or ' 

'}} 

{{=plugin_wiki . embed_page ( 

'meta-sidebar' 

> or 

"}} 

{{=plugin_wiki . embed_page ( 

'meta-footer' ) 

or ' 

'}} 
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In this way, the content of those pages will show up in the header, sidebar 
and footer in the layout. 


22.3.4 Configuring plugin_wiki 

As with any other plugins in "models/db.py" you can do 

1 from gluon. tools import PluginManager 

2 plugins = PluginManager( ) 

3 plugins. wiki. editor = auth. user. email == mail. settings. sender 

4 plugins. wiki. level = 3 

5 plugins. wiki. mode = 'markwin' or 'html' 

6 plugins. wiki. theme = ' ui-darkness' 

where 

• editor is true if the current logged-in user is authorized to edit plugin_wiki 
pages 

• level is the permission: 1 to edit regular pages, 2 to embed widgets in 
pages, 3 to embed code 

• mode determines whether to use a "markmin" editor or a WYSIWYG 
"html" editor. 

• theme is the name of the required jQuery UI Theme. By default only the 
color-neutral "ui-darkness" is installed. 

You can add themes here: 

1 static/plugin_wiki/ui/%( t heme )s/j query- ui- 1.8. 1. custom. ess 


12.3.5 Current widgets 

Each widget can be embedded both in plugin_wiki pages and in normal 
web2py templates. 

For example, to embed a YouTube video in a plugin_wiki page, you can do 

2 name: youtube 

3 code: l7AWnfFRc7g 

4 % % :widget 
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or to embed the same widget in a web2py view, you can do: 

{{=plugin_wiki. widget ( ' youtube' ,code=' l7AWnfFRc7g' )}} 

In either case, this is the output: 


BOO 


I 4 I ] [ + |>?http://lZ7.[).0.1.B00 Ci/myapp/plugin_wiki c] [Q* Google 


myapp 

example of a modular app 
Horns Search Engines Help Pages 


Embedded youtube video 


Record Updated 


RS A Animate -The Empathlc Civilisation 


HY 
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iwuvwuht IfttAWIC 
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lam C PWt 
NflflMiel 

W U fW f M wa 
ntfVAcr MHMt «T 

t^e «** <T. TM*V 

am?* TWfl Mn,M" 


& 





k> ■|.||| 00:00/10:40 igj II O 

Last modified by Massimo Di Pierro (1)1 secord ago [hi story] [edit] 


Copyright © 2010 - Powered by 


Widget arguments that do not have a default value are required. 
Here is a list of all current widgets: 

read 

read (table, record_id=None) 


Reads and displays a record 

• table is the name of a table 

• record-id is a record number 
create 


create ( table ,message= 
hidden_fields= 


, next=' ' , readonly_fields=' 
,default_fields=' ') 


COMPONENTS AND PLUGINS 5O5 

Displays a record create form 

• table is the name of a table 

• message is a the message to be displayed after record is created 

• next is where to redirect, example "page /index /[id]" 

• readonly_f ields is a list of comma separated fields 

• hidden_f ields is a list of comma separated fields 

• default-fields is a list of comma separated fieldname=value 

update 

update(table, record_id=' ' ,message=' ' ,next=' ' , 

readonly_fields=' ' , hidden_fields=' ' ,default_fields=' ' ) 

Displays a record update form 

• table is the name of a table 

• record_id is he record to be updated or {{= request . args(-l)}} 

• message is a the message to be displayed after record is created 

• next is where to redirect, example "page /index /[id]" 

• readonly-fields is a list of comma separated fields 

• hidden_fields is a list of comma separated fields 

• default_f ields is a list of comma separated fieldname=value 

select 

select(table,query_field=' ' ,query_value=' ' ,fields=' ' ) 

Lists all records in the table 

• table is the name of a table 

• query_field and query_value if present will filter records by according to 
the query query_field == query_value 

• fields is a list of comma separate fields to be displayed 
search 
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search (table, field s= 


Widgets for selecting records 

• table is the name of a table 

• fields is a list of comma separated fields to be displayed 

jqgrid 

jqgrid(table, fieldname=None,fieldvalue=None, col_widths=' ' , 

colnames=None,_id=None, fields^' ' ,col_width=80,width=700,height=30O) 

Embeds a jqGrid plugin 

• table is the table name 

• fieldname, f ieldvalue are an optional filter: f ieldname==f ieldvalue 

• col_widths is the width of each column 

• colnames is a list of column names that are displayed 

• _id is the "id" of the TABLE that contains the jqGrid 

• fields is a list of columns to be displayed 

• col_width is the default width of columns 

• height is the height of the jqGrid 

• width is the width of the jqGrid 

Once you have the plugin_wiki installed, you can easily use the jqGrid 
in your other views too. Example usage (displays yourtable filtered by 
fk_id==47): 

{{=plugin_wiki. widget ( 'jqgrid' , 'yourtable' , ' fk-id' ,47, ' 70, 150' , 
'Id, Comments' ,None, 'id, notes' ,80,300,200)}} 

latex 

latex(expression) 

Uses Google charting API to embed LaTeX 
pie_chart 
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pie_chart (data='I,2,3 ' , names='a,b, c' ,width=300 : height=150,align=' center' ) 


Embed a pie chart 

• data is a list of comma separated values 

• names is a list of comma separated labels (one for data item) 

• width is the width of the image 

• height is the height of the image 

• align determines the alignment of the image 

bar_chart 

bar_chart (data='l,2,3' , names='a,b, c' ,width=300 : height=150, align= 'center' ) 

Uses Google charting API to embed a bar chart 

• data is a list of comma separated values 

• names is a list of comma separated labels (one for data item) 

• width is the width of the image 

• height is the height of the image 

• align determines the alignment of the image 

slideshow 

slideshow(table : field= ' image ' , transition^ fade' , width=200 : height=200) 

Embeds a slideshow. It gets the images from a table. 

• table is the table name 

• field is the upload field in the table that contains images 

• transition determines the type of transition, e.g. fade, etc. 

• width is the width of the image 

• height is the height of the image 

youtube 

youtube(code : width=400, height=250) 
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Embeds a YouTube video (by code) 

• code is the code of the video 

• width is the width of the image 

• height is the height of the image 

vimeo 

vimeo(code, width=400, height=250) 

Embeds a Vimeo video (by code) 

• code is the code of the video 

• width is the width of the image 

• height is the height of the image 

mediaplayer 

mediaplayer(src : width=400, height=250) 

Embeds a media file (such as Flash video or an mp3 file) 

• src is the src of the video 

• width is the width of the image 

• height is the height of the image 

comments 

comments (table=' None' , record_id=None) 

Embeds comments in the page 

Comments can be linked to a table and /or a record 

• table is the table name 

• record_id is the id of the record 

tags 

tags(table='A/one' , record_id=None) 

Embeds tags in the page tags can be linked to a table and /or a record 
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• table is the table name 

• recorcLid is the id of the record 

tag_cloud 

tag_cloud() 

Embeds a tag cloud 

map 

map(key=' . . . . ' , table='auth_user' , width=400 : height=200) 

Embeds a Google map. 

It gets points on the map from a table 

• key is the google map api key (default works for 127.0.0.1) 

• table is the table name 

• width is the map width 

• height is the map height 

The table must have columns: latitude, longitude and map_popup. When 
clicking on a dot, the map_popup message will appear. 

iframe 

iframe(src, width=400, height=300) 

Embeds a page in an <if ramex/if rame> 

load_url 

load_url(src) 

Loads the content of the url using the LOAD function 

load_action 

load_action(action, controller^ ' , ajax=True) 

Loads the content of URL(request. application, controller, action) using the 
LOAD function 
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12.3.6 Extending widgets 

Widgets to plugin_wiki can be added by creating a new model file called 

"models /plugin_wiki_"nflme where name is arbitrary and the file contains 

something like: 

1 class PluginWikiWidgets(PluginWikiWidgets) : 

z @staticmethod 

3 def my_new_widget(argl, arg2=' value' , arg3=' value' ) : 

4 

5 document the widget 

6 

7 return "body of the widget" 

The first line states that you are extending the list of widgets. Inside the 
class, you can define as many functions as needed. Each static function is a 
new widget, except for functions that start with underscores. The function 
can take an arbitrary number of arguments which may or not have default 
values. The docstring of the function must document the function using the 
markmin syntax itself. 

When widgets are embedded into plugin_wiki pages, arguments will be 
passed to the widget as strings. This means the widget function must be 
able to accept strings for every argument and eventually convert them into 
the required representation. You can decide what the string representation 
must be - just make sure this is documented in the docstring. 

The widget can return a string of web2py helpers. In this latter case they will 
be serialized using . xml ( ) . 

Notice how the new widget can access any variable declared in the global 
namespace. 

As an example, we are going to create a new widget that display the 
"contact/ ask" form created at the beginning of this chapter. This can be done 
by creating a file "models /plugin_wiki_contact" that contains: 

1 class PluginWikiWidgets(PluginWikiWidgets) : 

2 @staticmethod 

, def ask(email_label=' Your email ' , question_label=' question' ) : 
4 

5 This plugin will display a contact us form that allows 

6 the visitor to ask a question. 
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The question will be emailed to you and the widget will 
disappear from the page. 
The arguments are 

- email_label: the label of the visitor email field 

- question-label: the label of the question field 


f rm=SQLFORM . facto ry ( 

Field( 'your-email ' , requires=IS_EMAIL( ) , label=email_label) , 
Field ( 'question' , requires=IS_NOT_EMPTY( ) ) , label=question_ label) 
if form. process( ) .accepted: 

if mail, send (to= ' adminisexample . com' , 

subject^' from %s' % form.vars.your_email, 
message = form. vars .question) : 
command="_/Query( '#%s' ) .hide( ) " % div_id 
response. flash = 'Thank you' 

response. js = "jQueryf '#%s' ) .hide( )" % request. cid 
else: 

form. errors. your_email="Unable to send the email" 
return form.xml() 

pluginjwiki widgets are not rendered by a view unless the 
response, renderf . . . ) function is called explicitly by the widget. 


13 

Deployment recipes 


There are multiple ways to deploy web2py in a production environment. The 
details depend on the configuration and the services provided by the host. 

In this chapter we consider the following issues: 

• Production deployment (Apache, Lighttpd, Cherokee) 

• Security 

• Scalability 

• Deployment on the Google App Engine platform(GAE [13] ) 

web2py comes with an SSL [21] enabled web server, the Rocket 
wsgiserver [22]. While this is a fast web server, it has limited configuration 
capabilities. For this reason it is best to deploy web2py behind Apache [82], 
Lighttpd [89] or Cherokee [90]. These are free and open-source web servers 
that are customizable and have been proven to be reliable in high traffic 
production environments. They can be configured to serve static files directly, 
deal with HTTPS, and pass control to web2py for dynamic content. 

Until a few years ago, the standard interface for communication between 
web servers and web applications was the Common Gateway Interface 
(CGI) [81]. The main problem with CGI is that it creates a new process 
for each HTTP request. If the web application is written in an interpreted 
language, each HTTP request served by the CGI scripts starts a new instance 
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of the interpreter. This is slow, and it should be avoided in a production 
environment. Moreover, CGI can only handle simple responses. It cannot 
handle, for example, file streaming. web2py provides a file cgihandler.py to 
interface to CGI. 

One solution to this problem is to use the mod_python module for Apache. 
We discuss it here because its use is still very common, though the 
mod_python project has officially been abandoned by the Apache Software 
Foundation. mod_python starts one instance of the Python interpreter when 
Apache starts, and serves each HTTP request in its own thread without 
having to restart Python each time. This is a better solution than CGI, 
but it is not an optimal solution, since mod_python uses its own interface 
for communication between the web server and the web application. In 
mod_python, all hosted applications run under the same user-id /group-id, 
which presents security issues. web2py provides a file modpythonhandler.py to 
interface to mod_python. 

In the last few years, the Python community has come together behind a 
new standard interface for communication between web servers and web 
applications written in Python. It is called Web Server Gateway Interface 
(WSGI) [17, 18]. web2py was built on WSGI, and it provides handlers for 
using other interfaces when WSGI is not available. 

Apache supports WSGI via the module mod_wsgi [88] developed by Graham 
Dumpleton. web2py provides a file wsgihandler.py to interface to WSGI. 

Some web hosting services do not support mod_wsgi. In this case, we must 
use Apache as a proxy and forward all incoming requests to the web2py 
built-in web server (running for example on localhost:8ooo). 

In both cases, with mod_wsgi and /or mod_proxy, Apache can be configured 
to serve static files and deal with SSL encryption directly, taking the burden 
off web2py 

The Lighttpd web server does not currently support the WSGI interface, but it 
does support the FastCGI [91] interface, which is an improvement over CGI. 
FastCGI's main aim is to reduce the overhead associated with interfacing 
the web server and CGI programs, allowing a server to handle more HTTP 
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requests at once. 

According to the Lighttpd web site, "Lighttpd powers several popular Web 
2.0 sites such as YouTube and Wikipedia. Its high speed IO-infrastructure 
allows them to scale several times better with the same hardware than 
with alternative web-servers". Lighttpd with FastCGI is, in fact, faster than 
Apache with mod_wsgi. web2py provides a file fcgihandler.py to interface to 
FastCGI. web2py also includes a gaehandler.py to interface with the Google 
App Engine (GAE). On GAE, web applications run "in the cloud". This means 
that the framework completely abstracts any hardware details. The web 
application is automatically replicated as many times as necessary to serve 
all concurrent requests. Replication in this case means more than multiple 
threads on a single server; it also means multiple processes on different 
servers. GAE achieves this level of scalability by blocking write access to 
the file system, and all persistent information must be stored in the Google 
BigTable datastore or in memcache. 

On non-GAE platforms, scalability is an issue that needs to be addressed, and 
it may require some tweaks in the web2py applications. The most common 
way to achieve scalability is by using multiple web servers behind a load- 
balancer (a simple round robin, or something more sophisticated, receiving 
heartbeat feedback from the servers). 

Even if there are multiple web servers, there must be one, and only one, 
database server. By default, web2py uses the file system for storing sessions, 
error tickets, uploaded files, and the cache. This means that in the default 
configuration, the corresponding folders have to be shared folders: 
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ggggg 


/ 


\ 




Shared Folder 

Samba or NFS 
(sessions, errors, cache, uploads) 

In the rest of the chapter, we consider various recipes that may provide an 
improvement over this naive approach, including: 

• Store sessions in the database, in cache or do not store sessions at all. 

• Store tickets on local filesystems and move them into the database in 
batches. 

• Use memcache instead of cache. ram and cache. disk. 

• Store uploaded files in the database instead of the shared filesystem. 

While we recommend following the first three recipes, the fourth recipe 
may provide an advantage mainly in the case of small files, but may be 
counterproductive for large files. 


1J.0.J anyserver.py 

tornado :inxx twisted :inxx wsgiref 

Web2py comes with a file called anyserver.py that implements WSGI 
interfaces to the following popular servers: bjoern, cgi, cherrypy, diesel, 
eventlet, fapws, flup, gevent, gnuicorn, mongrel2, paste, rocket, tornado, 
twisted, wsgiref 

You can use any of these servers, for example Tornado, simply by doing: 
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python anyserver.py -s tornado -i 127.0.0.1 -p 8000 -I -P 

( - 1 for logging and - P for profiler. For information on all the command line 
options use "-h": 

python anyserver.py -h 


23.2 Linux and Unix 


13.1.1 One step production deployment 

Here are some steps to install apache+python+mod_wsgi+web2py+postgresql 
from scratch. 

On Ubuntu: 

1 wget http://web2py.googlecode. com/hg/scripts/setup-web2py- ubuntu. sh 

2 chmod +x setup-web2py-ubuntu. sh 

3 sudo ./setup-web2py-ubuntu. sh 

On Fedora: 

1 wget http://web2py.googlecode. com/hg/sc ripts/ set up -web2py- fedora. sh 

2 chmod +x setup-web2py-fedora. sh 

3 sudo ./setup-web2py-fedora. sh 

Both of these scripts should run out of the box, but every Linux installation is 
a bit different, so make sure you check the source code of these scripts before 
you run them. In the case of Ubuntu, most of what they do is explained 
below. They do not implement the scalability optimizations discussed below. 


13.1.2 Apache setup 

In this section, we use Ubuntu 8.04 Server Edition as the reference platform. 
The configuration commands are very similar on other Debian-based Linux 
distribution, but they may differ for Fedora-based systems (which uses yum 
instead of apt -get). 
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First, make sure all the necessary Python and Apache packages are installed 
by typing the following shell commands: 



Then, enable the SSL module, the proxy module, and the WSGI module in 

Apache: 

] sudo In -s /etc/apache2/mods-available/proxy_http.load \ 
z /etc/apache2/mods-enabled/proxy_http.load 

3 sudo a2enmod ssl 

4 sudo a2enmod proxy 

5 sudo a2enmod proxy_http 

6 sudo a2enmod wsgi 

Create the SSL folder, and put the SSL certificates inside it: 

] sudo mkdir /etc/apache2/ssl 

You should obtain your SSL certificates from a trusted Certificate Authority 
such as verisign.com, but, for testing purposes, you can generate your own 
self-signed certificates following the instructions in ref. [87] 

Then restart the web server: 

1 sudo /etc/init .d/apache2 restart 

The Apache configuration file is: 

■ /etc/apache2/ sites -available/default 

The Apache logs are in: 
1 /var/log/apache2/ 


13. 1.3 mod_wsgi 

Download and unzip web2py source on the machine where you installed the 
web server above. 
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Install weD2py under /users/www- data/, for example, and give ownership to 
user www-data and group www-data. These steps can be performed with 
the following shell commands: 

1 cd /users/www-data/ 

2 sudo wget http://web2py.com/examples/static/web2py_src.zip 

3 sudo unzip web2py_src.zip 

4 sudo chown -R www-data :www-data /user/www-data/web2py 

To set up weD2py with mod_wsgi, create a new Apache configuration file: 

■ /et c/apache2/ sites -available/web2py 

and include the following code: 

1 <VirtualHost *:80> 

2 ServerName web2py.example.com 

i WSGIDaemonProcess web2py user=www-data group=www-data \ 

4 display-name=%{GROUP} 

5 WSGIProcessGroup web2py 

6 WSGIScriptAlias / /users/www-data/web2py/wsgihandler.py 
7 

s <Directory /users/www-data/web2py> 

i) AllowOverride None 

10 Order Allow, Deny 

u Deny from all 

12 <Files wsgihandler.py> 

13 Allow from all 

14 </Files> 

[ 5 </Directory> 
16 

17 AliasMatch A /( [V]+)/static/( .*) \ 

18 /users/www-data/web2py/applicat ion s/$l/ stat ic/$2 
10 <Directory /users/www-data/web2py/applications/*/static/> 

20 Order Allow, Deny 

21 Allow from all 

22 </Directory> 
23 

24 <Location /admin> 

2^ Deny from all 

26 </Location> 

27 

2s <LocationMatch V( [^/] + )/appadmin> 

21) Deny from all 

30 </LocationMatch> 

31 

32 CustomLog /private/var/log/apache2/access .log common 

33 ErrorLog /private/var/log/apache2/error.log 

34 </VirtualHost> 
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When you restart Apache, it should pass all the requests to web2py without 
going through the Rocket wsgiserver. 

Here are some explanations: 

WSGIDaemonProcess web2py user=www-data group=www-data 
display -name=%{GROUP} 

defines a daemon process group in context of "web2py.example.c0m". By 
defining this inside of the virtual host, only this virtual host can access this 
using WSGIProcessGroup, including any virtual host with the same server 
name but on a different port. The "user" and "group" options should be 
set to the user who has write access to the directory where web2py was 
setup. You do not need to set "user" and "group" if you made the web2py 
installation directory writable by the default user that Apache runs as. The 
"display-name" option makes the process name appears in ps output as 
"(wsgi:web2py)" instead of as name of Apache web server executable. As 
no "processes" or "threads" options are specified, the daemon process group 
will have a single process with 15 threads running within that process. This 
is usually more than adequate for most sites and should be left as is. If 
overriding it, do not use "processes=i" as doing so will disable any in- 
browser WSGI debugging tools that check the "wsgi. multiprocess" flag. This 
is because any use of the "processes" option will cause that flag to be set to 
true, even a single process, and such tools expect that it be set to false. Note: 
if your application code or third party extension module is not thread safe, 
use options "processes=5 threads=i" instead. This will create five processes in 
the daemon process group where each process is single threaded. You might 
consider using "maximum-requests=iooo" if your application leaks Python 
objects because it is unable to garbage collect properly. 

WSGIProcessGroup web2py 

delegates running of all WSGI applications to the daemon process group that 
was configured using the WSGIDaemonProcess directive. 

WSGIScriptAlias / /users/www-data/web2py/wsgihandler.py 

mounts the web2py application. In this case it is mounted at the root of the 
web site. 
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<Di rectory /users/www-data/web2py> 
</Directory> 


gives Apache permission to access the WSGI script file. 

<Di rectory /users/www-data/web2py/applications/*/static/> 

Order Allow, Deny 

Allow from all 
</Directory> 


Instructs Apache to bypass weD2py when searching static files. 

i <Location /admin> 

2 Deny from all 

3 </Location> 


and 

<Locationl*latch "/( [' s /] + )/appadmin> 

Deny from all 
</LocationMatch> 


blocks public access to admin and appadmin 

Normally we would just allow permission to the whole directory where 
the WSGI script file is located, but web2py places the WSGI script file in 
a directory which contains other source code, including the admin interface 
password. Opening up the whole directory would cause security issues, 
because technically Apache would be given permission to serve all the files 
up to any user who traversed to that directory via a mapped URL. To avoid 
security problems, explicitly deny access to the contents of the directory, 
except for the WSGI script file, and prohibit a user from doing any overrides 
from a.htaccess file to be extra safe. 

You can find a completed, commented, Apache wsgi configuration file in: 

script s/web2py-wsgi.conf 

This section was created with help from Graham Dumpleton, developer of 
mod_wsgi. 
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13.1.4 mod_wsgi and SSL 

To force some applications (for example admin and appadmin) to go over 
HTTPS, store the SSL certificate and key files: 

1 /etc/apache2/ssl/server.crt 

2 /etc/apache2/ssl/server.key 

and edit the Apache configuration file web2py.conf and append: 

■ <VirtualHost *:443> 

2 ServerName web2py.example.com 

3 SSLEngine on 

4 SSLCertificateFile /etc/apache2/ssl/server. crt 

5 SSLCertificateKeyFile /etc/apache2/ssl/server. key 

6 

7 WSGIProcessGroup web2py 

8 

i, WSGIScriptAlias / /users/www-data/web2py/wsgihandler.py 

ip <Directory /users/www-data/web2py> 

12 AllowOverride None 

13 Order Allow, Deny 

14 Deny from all 

15 <Files wsgihandler.py> 

16 Allow from all 

17 </Files> 

[8 </Directory> 
19 

AliasMatch A /( [~/]+)/static/( .*) \ 

21 /users/www-data/web2py/applications/$l/static/$2 
22 

21 <Directory /users/www-data/web2py/applications/*/static/> 

24 Order Allow, Deny 

25 Allow from all 
2b </Di recto ry> 

27 

28 CustomLog /private/var/log/apache2/access .log common 

2i, ErrorLog /private/var/log/apache2/error.log 

30 

31 </VirtualHost> 


Restart Apache and you should be able to access: 

, https://www.example.com/admin 

2 https://www.example.com/examples/appadmin 

3 http://www.example.com/examples 


DEPLOYMENT RECIPES 523 


but not: 

http : //www . example . com/admin 
http://www.example.com/examples/appadmin 


13.1.5 modjproxy 

Some Unix /Linux distributions can run Apache, but do not support 
mod_wsgi. In this case, the simplest solution is to run Apache as a proxy 
and have Apache deal with static files only. 

Here is a minimalist Apache configuration: 

NameVirtualHost *:80 

### deal with requests on port 80 

<VirtualHost *:80> 

Alias / /users/www-data/web2py/applications 
### serve static files directly 
<LocationMatch ""/welcome/ static/ . *"> 
Order Allow, Deny 
Allow from all 
</LocationMatch> 

### proxy all the other requests 
<Location "/welcome"> 
Order deny.allow 
Allow from all 
ProxyRequests off 

ProxyPass http://localhost :8000/welcome 
ProxyPassReverse http : //localhost : 8000/ 
ProxyHTMLURLMap http ://127. 0.0. l:8000/welcome/ /welcome 
</Location> 

LogFormat "%h %l %u %t "%r" %>s %£>" common 
CustomLog /var/log/apache2/access.log common 
</VirtualHost> 


The above script exposes only the "welcome" application. To expose other 
applications, you need to add the corresponding <Location>...</Location> 
with the same syntax as done for the "welcome" app. 

The script assumes there is a web2py server running on port 8000. Before 
restarting Apache, make sure this is the case: 

nohup python web2py.py -a '<recycle>' -i 127.0.0.1 -p 8000 & 
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You can specify a password with the -a option or use the "<recycle>" 
parameter instead of a password. In the latter case, the previously stored 
password is reused and the password is not stored in the shell history. 

You can also use the parameter "<ask>", to be prompted for a password. 

The nohup commands makes sure the server does not die when you close the 
shell, nohup logs all output into nohup. out. 

To force admin and appadmin over HTTPS use the following Apache 
configuration file instead: 

1 NameVirtualHost *:80 

2 NameVirtualHost *:443 

3 ### deal with requests on port 80 

4 <VirtualHost *:80> 

5 Alias / /usres/www-data/web2py/applications 

6 ### admin requires SSL 

- <LocationMatch "~/admin"> 

s SSLRequireSSL 

i) </LocationMatch> 

[o ### appadmin requires SSL 

i] <LocationMatch " "/welcome/ appadmin / .*"> 

12 SSLRequireSSL 

13 </LocationMatch> 

i 4 ### serve static files directly 

i- <LocationMatch ""/welcome/ static/ .*"> 

16 Order Allow, Deny 

i 7 Allow from all 

is </LocationMatch> 

i 9 ### proxy all the other requests 

20 <Location "/welcome"> 

21 Order deny.allow 

22 Allow from all 

23 ProxyPass http://localhost :8000/welcome 

24 ProxyPassReverse http://localhost :8000/ 
-^ </Location> 

LogFormat "%/? %l %u %t "%r" %>s %h" common 
27 CustomLog /var/log/apache2/access.log common 

is </VirtualHost> 

2o <VirtualHost *:443> 
3 o SSLEngine On 
3 i SSLCertificateFile /etc/apache2/ssl/server. crt 

32 SSLCertificateKeyFile /etc/apache2/ssl/server. key 

33 <Location "/"> 

34 Order deny.allow 

35 Allow from all 

3 6 ProxyPass http://localhost :8000/ 
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37 ProxyPassReverse http://localhost :8000/ 

38 </Location> 

LogFormat "%/7 %l %u %t "%r" %>s %b" common 
4 n CustomLog /var/log/apache2/access.l.og common 
41 </VirtualHost> 

The administrative interface must be disabled when webzpy runs on a shared 
host with mod_proxy, or it will be exposed to other users. 

13.1.6 Start as Linux daemon 

Unless you are using mod_wsgi, you should setup the web2py server so that 
it can be started /stopped /restarted as any other Linux daemon, and so it can 
start automatically at the computer boot stage. 

The process to set this up is specific to various Linux /Unix distributions. 

In the web2py folder, there are two scripts which can be used for this 

purpose: 

1 scripts/web2py .ubuntu.sh 
z scripts/web2py . fedora. sh 

On Ubuntu, or other Debian-based Linux distribution, edit 
"web2py.ubuntu.sh" and replace the "/usr/lib/web2py" path with the 
path of your web2py installation, then type the following shell commands to 
move the file into the proper folder, register it as a startup service, and start 
it: 

1 sudo cp scripts/web2py. ubuntu . sh /etc/init .d/web2py 

2 sudo update- red web2py defaults 

3 sudo /etc/init .d/web2py start 

On Fedora, or any other distributions based on Fedora, edit 
"web2py.fed0ra.sh" and replace the "/usr/lib/web2py" path with the 
path of your web2py installation, then type the following shell commands to 
move the file into the proper folder, register it as a startup service and start 
it: 

1 sudo cp scripts/web2py. fedora. sh /etc/rc.d/init .d/web2pyd 

2 sudo chkeonfig --add web2pyd 

3 sudo service web2py start 


526 WEB2PY PULL-STACK WEB FRAMEWORK, 4TH EDITION 


13.1.7 Lighttpd 

You can install Lighttpd on a Ubuntu or other Debian-based Linux 
distribution with the following shell command: 

apt-get -y install lighttpd 

Once installed, edit /etc/ re. local and create a fcgi web2py background 
process 

cd /var/www/web2py && sudo -u www-data nohup python fcgihandler.py & 

Then, you need to edit the Lighttpd configuration file 

/etc/lighttpd/lighttpd . conf 

so that it can find the socket created by the above process. In the config file, 
write something like: 

server. modules = ( 

"mod-access" , 
"mod-alias" , 
"mod-compress" , 
"mod-rewrite" , 
"mod-fastcgi" , 
"mod-redirect" , 
"mod-accesslog" , 
"modstatus" , 


... ) 


iz server. port = 80 

13 server. bind = "0.0.0.0" 

i 4 server. event-handler = "freebsd-kqueue" 

15 server. error-handler-404 = "/test . fcgi" 

16 server. document- root = "/users/www-data/web2py/" 
i 7 server. errorlog = "/tmp/error.log" 

18 

ii) fastcgi. server = ( 

20 "/handler-web2py.fcgi" => ( 

21 " handle r_web2py" => ( #name for logs 

22 "check-local" => "disable" , 

23 "socket" => "Itmpl fcgi. sock" 


2s $HTTP["host"] = "("\\.)example\.com$" { 
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29 server. document- root-" /var/www/web2py" 

30 url. rewrite-once = ( 

3 i ""(/ .+?/static/ .+)$" => "/applications$l" 

32 " ("\/ .*)$" => "/handler^web2py.fcgi$l", 

33 ) 

34 } 


Now check for syntax errors: 

lighttpd -t -f /etc/Ughttpd/lighttpd.conf 

and (re)start the web server with: 
/etc/init .d/lighttpd restart 

Notice that FastCGI binds the web2py server to a Unix socket, not to an IP 
socket: 

/tmp/fcgi.sock 

This is where Lighttpd forwards the HTTP requests to and receives responses 
from. Unix sockets are lighter than Internet sockets, and this is one of the 
reasons Lighttpd+FastCGI+web2py is fast. As in the case of Apache, it is 
possible to setup Lighttpd to deal with static files directly, and to force some 
applications over HTTPS. Refer to the Lighttpd documentation for details. 

Examples in this section were taken from John Heenan's post in web2pyslices. 

The administrative interface must be disabled when webzpy runs on a shared 
host with FastCGI, or it will be exposed to the other users. 


13.1.8 Shared hosting with mod_python 

There are times, specifically on shared hosts, when one does not have the 
permission to configure the Apache config files directly. At the time of 
writing most of these hosts still run mod_python even if it is not maintained 
any more in favor of mod_wsgi. 

You can still run web2py. Here we show an example of how to set it up. 

Place contents of web2py into the "htdocs" folder. 
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In the web2py folder, create a file "web2py_modpython.py" file with the 
following contents: 

1 from mocLpython import apache 

2 import modpythonhandler 

3 

4 def handler( req) : 

5 req. subprocess_env[ 'PATH-INFO' ] = req. subprocess_env[ 'SCRIPT-URL ' ] 

6 return modpythonhandler. handler(req) 

Create/update the file ".htaccess" with the following contents: 

1 SetHandler python-program 

z PythonHandler web2py_modpython 

3 #PythonDebug On 


This example was provided by Niktar. 


13.1.9 Cherokee with FastCGI 

Cherokee is a very fast web server and, like web2py, it provides an AJAX- 
enabled web-based interface for its configuration. Its web interface is written 
in Python. In addition, there is no restart required for most of the changes. 

Here are the steps required to setup web2py with Cherokee: 

Download Cherokee [90] 

Untar, build, and install: 

1 tar -xzf cherokee-0.9.4.tar.gz 

2 cd cherokee-0.9.4 

3 ./configure - -enable-fcgi && make 

4 make install 

Start web2py normally at least once to make sure it creates the "applications" 
folder. 

Write a shell script named "startweb2pysh" with the following code: 

, #!/bin/bash 

2 cd /var/web2py 

3 python /var/web2py/fcgihandler. py & 
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and give the script execute privileges and run it. This will start weD2py under 
FastCGI handler. 

Start Cherokee and cherokee-admin: 

1 sudo nohup Cherokee & 

2 sudo nohup cherokee-admin & 

By default, cherokee-admin only listens at local interface on port 9090. This 
is not a problem if you have full, physical access on that machine. If this is 
not the case, you can force it to bind to an IP address and port by using the 
following options: 

, -b, --bind[=IP] 
a -p, --port=NUM 

or do an SSH port-forward (more secure, recommended): 
, ssh -L 9090 :localhost: 9090 remotehost 

Open "http://localhost:909o" in your browser. If everything is ok, you will 
get cherokee-admin. 

In cherokee-admin web interface, click "info sources". Choose "Local 
Interpreter". Write in the following code, then click "Add New". 

1 Nick: web2py 

2 Connection: /tmp/fcgi.sock 

3 Interpreter: /var/web2py/startweb2py.sh 

Finally, perform the following remaining steps: 

• Click "Virtual Servers", then click "Default". 

• Click "Behavior", then, under that, click "default". 

• Choose "FastCGI" instead of "List and Send" from the list box. 

• At the bottom, select "web2py" as "Application Server" 

• Put a check in all the checkboxes (you can leave Allow-x-sendfile). If there 
is a warning displayed, disable and enable one of the checkboxes. (It will 
automatically re-submit the application server parameter. Sometimes it 
doesn't, which is a bug). 
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• Point your browser to "http://yoursite", and "Welcome to web2py" will 
appear. 

13.1.10 Postgresql 

PostgreSQL is a free and open source database which is used in demanding 
production environments, for example, to store the.org domain name 
database, and has been proven to scale well into hundreds of terabytes of 
data. It has very fast and solid transaction support, and provides an auto- 
vacuum feature that frees the administrator from most database maintenance 
tasks. 

On an Ubuntu or other Debian-based Linux distribution, it is easy to install 

PostgreSQL and its Python API with: 

sudo apt-get -y install postgresql 
sudo apt-get -y install python-psycopg2 

It is wise to run the web server(s) and the database server on different 
machines. In this case, the machines running the web servers should be 
connected with a secure internal (physical) network, or should establish SSL 
tunnels to securely connect with the database server. 

Edit the PostgreSQL configuration file 

sudo nano /etc/postgresql/8.4/main/postgresql . conf 

and make sure it contains these two lines 

track_counts = on 

autovacuum = on # Enable autovacuum subprocess? 'on' 

Start the database server with: 
sudo /etc/init .d/postgresql restart 

When restarting the PostgreSQL server, it should notify which port it is 
running on. Unless you have multiple database servers, it should be 5432. 


The PostgreSQL logs are in: 
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/var/log/postgresql/ 


Once the database server is up and running, create a user and a database so 
that web2py applications can use it: 



The first of the commands will grant superuser-access to the new user, called 
myuser. It will prompt you for a password. 

Any web2py application can connect to this database with the command: 
db = DAL( "postgres://myuser:mypassword@localhost:5432/mydb" ) 

where mypassword is the password you entered when prompted, and 5432 is 
the port where the database server is running. 

Normally you use one database for each application, and multiple instances 
of the same application connect to the same database. It is also possible for 
different applications to share the same database. 

For database backup details, read the PostgreSQL documentation; 
specifically the commands pg_dump and pg_ restore. 


13.2 Windows 

13.2.1 Apache and mod_wsgi 

Installing Apache, and mod_wsgi under Windows requires a different 
procedure. Here are assuming Python 2.5 is installed, you are running from 
source and web2py is located at c : /web2py. 

First download the requires packages: 

• Apache apache_2.2.11-win32-x86-openssl-Q.9.8i.msi from [83] 

• mod_wsgi from [84] 
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Second, run apache. . .msi and follow the wizard screens. On the server 
information screen 


\z Apache HTTP Se i ver 1.1 - Installs ion Wizard 


Server Information 

Please enter your server's information. 


Network Domain (e.g. somenet.com) 


Server Name (e.g. wwv.somenet.com): 


Adrwilstiator's Email ftddrftss [e.g. tvebm«l«r®«orMn§t.com)! 


Install Apache HTTP Server 2.2 programs and shortcuts for: 

f* for £JI Users, on Port 80, as a Service ~ Recommended- 

C only for the Current User, on Port 8080, wtien started Manually. 


enter all requested values: 

• Network Domain: enter the DNS domain in which your server is or 
will be registered in. For example, if your server's full DNS name is 
server.mydomain.net, you would type mydomain.net here 

• ServerName: Your server's full DNS name. From the example above, 
you would type server.mydomain.net here. Enter a fully qualified domain 
name or IP address from the web2py install, not a shortcut, for more 
information see [86]. 

• Administrator's Email Address. Enter the server administrator's or 
webmaster's email address here. This address will be displayed along 
with error messages to the client by default. 

Continue with a typical install to the end unless otherwise required 

The wizard, by default, installed Apache in the folder: 

C:/Program Files/Apache Software Foundation/Apache2.2/ 


From now on we refer to this folder simply as Apache2 . 2. 

Third, copy the downloaded mod_wsgi.so to Apache2.2/modules written by 
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Chris Travers, published by the Open Source Software Lab at Microsoft, 
December 2007. 

Fourth, create server. crt and server, key certificates (as discussed in the 
previous section) and place them in the folder Apache2.2/conf. Notice the 
cnf file is in Apache2 . 2/conf /openssl . cnf . 

Fifth, edit Apache2. 2/conf /httpd. conf, remove the comment mark (the # 
character) from the line 

LoadPlodule ssl_module modules/mod_ssl.so 

add the following line after all the other LoadModule lines 

LoadPlodule wsgi_module modules/mod_wsgi.so 


look for "Listen 80" and add this line after it 

1 Listen 443 


append the following lines at the end changing drive letter, port number, 
ServerName according to your values 


NameVirtualHost *:443 
<VirtualHost *:443> 

Document Root " C : /web2py / applications" 

ServerName serverl 

<Directory "C:/web2py"> 

Order allow, deny 

Deny from all 
</Directory> 

<Location "/"> 

Order deny.allow 

Allow from all 
</Location> 

<LocationMatch ""{/ [\W-]*/ static/ . *) "> 

Order Allow, Deny 

Allow from all 
</LocationMatch> 

WSGIScriptAlias / "C:/web2py/wsgihandler.py" 

SSLEngine On 

SSLCertificateFile conf /server. crt 
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SSLCertificateKeyFile 

conf/server. 

key 

LogFormat 

"%/i %I %u %t "%r" %>s %fc 

" common 

CustomLog 

logs/access. 

log common 


</VirtualHost> 




Save and check the config using: [Start > Program > Apache HTTP Server 2.2 
> Configure Apache Server > Test Configuration] 

If there are no problems you will see a command screen open and close. Now 
you can start Apache: 

[Start > Program > Apache HTTP Server 2.2 > Control Apache Server > Start] 
or better yet start the taskbar monitor 

[Start > Program > Apache HTTP Server 2.2 > Control Apache Server] 

Now you can right-click on the red feather-like taskbar icon to "Open Apache 
Monitor" and then start, stop and restart Apache as required. 

This section was created by Jonathan Lundell. 


13.2.2 Start as Windows service 

What Linux calls a daemon, Windows calls a service. The web2py server can 
easily be installed /started /stopped as a Windows service. 

In order to use web2py as a Windows service, you must create a file 
"options.py" with startup parameters: 

1 import socket, os 

2 ip = socket .gethostname( ) 

3 port = 80 

4 password = '<recycle>' 

5 picLfilename = ' httpserver.pid' 

6 log_filename = 'httpserver.log' 

7 ssl_certificate = " 
s ssl-private-key = " 
9 numthreads = 10 

id server_name = socket .gethostname( ) 

11 request_queue_size = 5 

12 timeout = 10 

13 shutdown_timeout = 5 

14 folder = os.getcwd() 


DEPLOYMENT RECIPES 535 


You don't need to create "options. py" from scratch since there is already an 
"options_stdpy" in the weD2py folder that you can use as a model. 

After creating "optionspy" in the weD2py installation folder, you can install 
weD2py as a service with: 
python web2py.py -W install 

and start/stop the service with: 

python web2py.py -W start 
python web2py.py -W stop 


23.3 Securing sessions and admin 

It is very dangerous to publicly expose the admin application and the 
appadmin controllers unless they run over HTTPS. Moreover, your password 
and credentials should never be transmitted unencrypted. This is true for 
web2py and any other web application. 

In your applications, if they require authentication, you should make the 
session cookies secure with: 
session. secure( ) 

An easy way to setup a secure production environment on a server is to 
first stop web2py and then remove all the parameters_*.py files from the 
web2py installation folder. Then start web2py without a password. This 
will completely disable admin and appadmin. 

nohup python web2py --nogui -p 8001 -i 127.0.0.1 -a ' ' & 

Next, start a second web2py instance accessible only from localhost: 

nohup python web2py --nogui -p 8002 -i 127.0.0.1 -a '<ask>' & 

and create an SSH tunnel from the local machine (the one from which you 
wish to access the administrative interface) to the server (the one where 
web2py is running, example.com), using: 

ssh -L 8002:127.0.0.1:8002 username@example.com 
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Now you can access the administrative interface locally via the web browser 
at localhost:8002. 

This configuration is secure because admin is not reachable when the tunnel 
is closed (the user is logged out). 

This solution is secure on shared hosts if and only if other users do not have 
read access to the folder that contains webzyy; otherwise users may be able to 
steal session cookies directly from the server. 

13.4 Efficiency and scalability 

web2py is designed to be easy to deploy and to setup. This does not mean 
that it compromises on efficiency or scalability but it means you may need 
to tweak it to make it scalable. 

In this section we assume multiple web2py installations behind a NAT server 
that provides local load-balancing. 

In this case, web2py works out-of-the-box if some conditions are met. In 

particular, all instances of each web2py application must access the same 

database servers and must see the same files. This latter condition can be 

implemented by making the following folders shared: 

■ applications/myapp/sessions 

z applications/myapp/errors 

3 applications/myapp/uploads 

4 applications/myapp/cache 

The shared folders must support file locking. Possible solutions are ZFS 
(ZFS was developed by Sun Microsystems and is the preferred choice.), NFS 
(With NFS you may need to run thenlockmgr daemon to allow file locking.), 
or Samba (SMB). 

It is possible to share the entire web2py folder or the entire applications 
folder, but this is not a good idea because this would cause a needless 
increase of network bandwidth usage. 

We believe the configuration discussed above to be very scalable because 
it reduces the database load by moving to the shared filesystems those 
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resources that need to be shared but do not need transactional safety (only 
one client at a time is supposed to access a session file, cache always needs a 
global lock, uploads and errors are write once /read many files). 

Ideally, both the database and the shared storage should have RAID 
capability. Do not make the mistake of storing the database on the same 
storage as the shared folders, or you will create a new bottleneck there. 

On a case-by-case basis, you may need to perform additional optimizations 
and we will discuss them below. In particular, we will discuss how to get 
rid of these shared folders one-by-one, and how to store the associated data 
in the database instead. While this is possible, it is not necessarily a good 
solution. Nevertheless, there may be reasons to do so. One such reason is 
that sometimes we do not have the freedom to set up shared folders. 

23.4.1 Efficiency tricks 

web2py application code is executed on every request, so you want to 
minimize this amount of code. Here is what you can do: 

• Run once with migrate=True then set all your tables to migrate=False. 

• Bytecode compile your app using admin. 

• Use cache . ram as much as you can but make sure to use a finite set of keys, 
or else the amount of cache used will grow arbitrarily. 

• Minimize the code in models: do not define functions there, define 
functions in the controllers that need them or - even better - define 
functions in modules, import them and use those functions as needed. 

• Do not put many functions in the same controller but use many controllers 
with few functions. 

• Call session. forget( response) in all controllers and/or functions that do 
not change the session. 

• Try to avoid web2py cron, and use a background process instead. web2py 
cron can start too many Python instances and cause excessive memory 
usage. 
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23.4.2 Sessions in database 

It is possible to instruct web2py to store sessions in a database instead of 
in the sessions folder. This has to be done for each individual web2py 
application, although they may all use the same database to store sessions. 

Given a database connection 

db = DAL(. . .) 

you can store the sessions in this database (db) by simply stating the 
following, in the same model file that establishes the connection: 
session. connect (request, response, db) 

If it does not exist already, web2py creates, under the hood, a table in the 

database called web2py_session_flppname containing the following fields: 

Field (' locked ' , 'boolean', default=False) 

Field ( ' client-ip' ) , 

Field (' created_datetime' , 'datetime', default=now) , 

Field ( ' modified-datetime' , 'datetime' 

Field ( ' unique_key ' ) , 

Field (' session-data' , 'text') 

"unique_key" is a uuid key used to identify the session in the cookie. 
"session_data" is the cPickled session data. 

To minimize database access, you should avoid storing sessions when they 
are not needed with: 
session. forget ( ) 

With this tweak the "sessions" folder does not need to be a shared folder 
because it will no longer be accessed. 

Notice that, if sessions are disabled, you must not pass the session to 
form. accepts and you cannot use session. flash nor CRUD. 

13.4.3 HAProxy a high availability load balancer 

If you need multiple web2py processes running on multiple machines, 
instead of storing sessions in the database or in cache, you have the option to 
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use a load balancer with sticky sessions. 

Pound [92] and HAProxy [93] are two HTTP load balancers and Reverse 
proxies that provides sticky sessions. Here we discuss the latter because it 
seems to be more common on commercial VPS hosting. 

By sticky sessions, we mean that once a session cookie has been issued, the 
load balancer will always route requests from the client associated to the 
session, to the same server. This allows you to store the session in the local 
filesystem without need for a shared filesystem. 

To use HAProxy: 

First, install it, on out Ubuntu test machine: 
sudo apt-get -y install haproxy 



Second edit the configuration file "/etc/haproxy.cfg" to something like this: 


# this config needs haproxy-1.1.28 or haproxy-1.2.1 

3 

global 

4 

log 127.0.0.1 local© 

5 

maxconn 1024 

6 

daemon 

7 
8 

defaults 

9 

log global 

10 

mode http 

.1 

option httplog 

12 

option httpchk 

13 

option httpclose 

14 

retries 3 

15 

option redispatch 

16 

contimeout 5000 

*7 

clitimeout 50000 

18 

srvtimeout 50000 

!9 

listen 0.0.0.0:80 

21 

balance url_param WEB2PYSTICKY 

22 

balance roundrobin 

23 

server Ll_l 10.211.55.1:7003 check 

24 

server Ll_2 10.211.55.2:7004 check 

25 

server Ll_3 10.211.55.3:7004 check 

26 

appsession WEB2PYSTICKY len 52 timeout lh 


The listen directive tells HAProxy, which port to wait for connection from. 
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The server directive tells HAProxy where to find the proxyed servers. The 
appsession directory makes a sticky session and uses the a cookie called 
WEB2PYSTICKY for this purpose. 

Third, enable this config file and start HAProxy: 

/etc/init .d/haproxy restart 

You can find similar instructions to setup Pound at the URL 

http://web2pyslices.com/main/slices/take_slice/33 


13.4.4 Cleaning up sessions 

You should be aware that on a production environment, sessions pile up fast. 
web2py provides a script called: 

scripts/sessions2trash.py 

that when run in the background, periodically deletes all sessions that have 
not been accessed for a certain amount of time. Web2py provides a script 
to cleanup these sessions (it works for both file-based sessions and database 
sessions). 

Here are some typical use cases: 

• Delete expired sessions every 5 minutes: 

nohup python web2py.py -S app -M -R scripts/sessions2trash.py _ 

• Delete sessions older than 60 minutes regardless of expiration, with 
verbose output, then exit: 

python web2py.py -S app -M -R scripts/sessions2trash.py -A -0 -x 3600 -f -v 

• Delete all sessions regardless of expiry and exit: 

python web2py.py -S app -M -R scripts/sessions2trash.py -A -0 -x 

Here app is the name of your application. 
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23.4.5 Uploading files in database 

By default, all uploaded files handled by SQLFORMs are safely renamed and 
stored in the filesystem under the "uploads" folder. It is possible to instruct 
web2py to store uploaded files in the database instead. 

Now, consider the following table: 

1 db.define_table( 'dog' , 

2 Field ( ' name ' ) 

3 Field (' image ' , 'upload')) 


where dog . image is of type upload. To make the uploaded image go in the 
same record as the name of the dog, you must modify the table definition by 
adding a blob field and link it to the upload field: 


db.define_table( 'dog' , 


Field ( ' name ' ) 


Field ( 'image' , 'upload', uploadfield= 

'image^data' ) , 

Field ( 'image-data ' , 'blob')) 



Here "image_data" is just an arbitrary name for the new blob field. 

Line 3 instructs web2py to safely rename uploaded images as usual, store 
the new name in the image field, and store the data in the uploadfield called 
"image_data" instead of storing the data on the filesystem. All of this is be 
done automatically by SQLFORMs and no other code needs to be changed. 

With this tweak, the "uploads" folder is no longer needed. 

On Google App Engine, files are stored by default in the database without 
the need to define an uploadfield, since one is created by default. 


13.4.6 Collecting tickets 

By default, web2py stores tickets (errors) on the local file system. It would 
not make sense to store tickets directly in the database, because the most 
common origin of error in a production environment is database failure. 

Storing tickets is never a bottleneck, because this is ordinarily a rare event. 
Hence, in a production environment with multiple concurrent servers, it is 
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more than adequate to store them in a shared folder. Nevertheless, since only 
the administrator needs to retrieve tickets, it is also OK to store tickets in a 
non-shared local "errors" folder and periodically collect them and/or clear 
them. 

One possibility is to periodically move all local tickets to the database. 

For this purpose, web2py provides the following script: 

script s/tickets2db.py 

By default the script gets the db uri from a file saved into the private folder, 

ticket_storage.txt. This file should contain a string that is passed directly to 

a DAL instance, like: 

mysql: //use rname:password(alocal host /test 
postgres://username:password<alocalhost/test 

This allows to leave the script as it is: if you have multiple applications, it 
will dynamically choose the right connection for every application. If you 
want to hardcode the uri in it, edit the second reference to db_string, right 
after the except line. You can run the script with the command: 
nohup python web2py.py -S myapp -M -R scripts/tickets2db. py & 

where myapp is the name of your application. 

This script runs in the background and moves all tickets every 5 minutes to 
a table and removes the local tickets. You can later view the errors using the 
admin app, clicking on the "switch to: db" button at the top, with the same 
exact functionality as if they were stored on the file system. 

With this tweak, the "errors" folder does not need to be a shared folder any 
more, since errors will be stored into the database. 


13. 4.7 Memcache 

We have shown that web2py provides two types of cache: cache, ram and 
cache. disk. They both work on a distributed environment with multiple 
concurrent servers, but they do not work as expected. In particular, cache . ram 
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will only cache at the server level; thus it becomes useless, cache. disk will 
also cache at the server level unless the "cache" folder is a shared folder that 
supports locking; thus, instead of speeding things up, it becomes a major 
bottleneck. 

The solution is not to use them, but to use memcache instead. web2py comes 
with a memcache API. 

To use memcache, create a new model file, for example 0_memcache . py, and in 
this file write (or append) the following code: 

1 from gluon. contrib. memcache import MemcacheClient 

2 memcache_servers = [' 127.0.0.1:11211' ] 

3 cache. memcache = MemcacheClient (request, memcache_servers) 

4 cache. ram = cache. disk = cache. memcache 


The first line imports memcache. The second line has to be a list of memcache 
sockets (server:port). The third line defines cache. memcache. The fourth line 
redefines cache, ram and cache. disk in terms of memcache. 

You could choose to redefine only one of them to define a totally new cache 
object pointing to the Memcache object. 

With this tweak the "cache" folder does not need to be a shared folder any 
more, since it will no longer be accessed. 

This code requires having memcache servers running on the local network. 
You should consult the memcache documentation for information on how to 
setup those servers. 


23.4.8 Sessions in memcache 

If you do need sessions and you do not want to use a load balancer with 
sticky sessions, you have the option to store sessions in memcache: 

1 from gluon. cont rib. memdb import MEMDB 

2 session. connect (request, response, db=MEMDB( cache. memcache) ) 
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23.4.9 Caching with Redis 

[103] An alternative to Memcache is use Redis. 

Assuming we have Redis installed and running on localhost at port 6379, we 
can connect to it using the following code (in a model): 

from gluon.contrib. redis import RedisCache 

cache. redis = RedisCache( ' localhost : 6379 ' ,db=None, debug=True) 

where 'localhost:6379' is the connection string and db is not a DAL object but 
a Redis database name. 

We can now use cache, redis in place of (or along with) cache, ram and 
cache. disk. 

We can also obtain Redis statistcs by calling: 
cache, redis .statsO 


13.4.10 Removing applications 

In a production setting, it may be better not to install the default applications: 
admin, examples and welcome. Although these applications are quite small, 
they are not necessary. 

Removing these applications is as easy as deleting the corresponding folders 
under the applications folder. 


13.4.11 Using replicated databases 

In a high performance environment you may have a master-slave database 
architecture with many replicated slaves and perhaps a couple of replicated 
servers. The DAL can handle this situation and conditionally connect to 
different servers depending on the request parameters. The API to do this 
was described in Chapter 6. Here is an example: 

] from random import sample 

a db = DAL ( sample (['mysq/l://. ■■!' , 'mysql://. . .2' , 'mysql://. . .3'], 3)) 
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In this case, different HTTP requests will be served by different databases at 
random, and each DB will be hit more or less with the same probability. 

We can also implement a simple Round-Robin 



This is fail-safe in the sense that if the database server assigned to the request 
fails to connect, DAL will try the next one in the order. 

It is also possible to connect to different databases depending on the 
requested action or controller. In a master-slave database configuration, some 
action performs only a read and some person both read/write. The former 
can safely connect to a slave db server, while the latter should connect to a 
master. So you can do: 

1 if request . function in read_only_actions: 

db = DAL(sample( ['mysql://. . .1' , 'mysql:// . . .2', 'mysql://. . .3'], 3)) 

3 if request .action in read_only_actions: 

4 db = DAL(shuffle( ['mysql://. ..1', 'mysql://. . .2', 'mysql://. . .3'])) 

5 else: 

6 db = DAL(sample( [ 'mysql://. . .3' , 'mysql://. . .4' , 'mysql://. . .5' ] , 3)) 

where 1,2,3 are slaves and 3,4,5 are masters. 


23.5 Deploying on Google App Engine 

It is possible to run web2py code on Google App Engine (GAE) [13], 
including DAL code. 

GAE supports two versions of Python: 2.5 (default) and 2.7 (beta). web2y 
supports both but uses 2.5 by default (this may change in the future). Look 
into the "app.yaml" file described below for configration details. 

GAE also supports a Google SQL database (compatible with MySQL) and a 
Google NoSQL (referred to as "Datastore"). web2py supports both. If you 
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wish to use Google SQL database follow the instructions on Chapter 6. This 
section assues you will be using the Google Datastore. 

The GAE platform provides several advantages over normal hosting 
solutions: 

• Ease of deployment. Google completely abstracts the underlying 
architecture. 

• Scalability. Google will replicate your app as many times as it takes to 
serve all concurrent requests. 

• One can choose between a SQL and a NoSQL database (or both together). 
But also some disadvantages: 

• No read or write access to the file system. 

• No HTTPS unless you use the appspot.com domain with a Google 
certificate. 

and some Datastore specific disadvantages: 

• No typical transactions. 

• No complex datastore queries. In particular there are no JOIN, LIKE, and 
DATE/DATETIME operators. 

• No multiple OR sub-queries unless they involve one and the same field. 

Because of the readonly filesystem, web2py cannot store sessions, error 
tickets, cache files and uploaded files in the filesystem; they must be stored 
in the datastore and not in the filesystem. 

Here we provide a quick overview of GAE and we focus on web2py specific 
issues, we refer you to the official GAE documentation online for details. 

Attention: At the time of writing, GAE supports only Python 2.5. Any other 
version will cause problems. You also must run the webzpy source distribution, 
not a binary distribution. 
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13.3.1 Configuration 

There are three configuration files to be aware of: 

1 web2py/app.yaml 

2 web2py/queue.yaml 
, web2py/index.yaml 

app.yaml and queue. yaml are most easily created by using the template files 
app. example. yaml and queue. example. yaml as starting points, index. yaml is 
created automatically by the Google deployment software. 

app.yaml has the following structure (it has been shortened using...): 

1 application: web2py 

2 version: 1 

3 api_version: 1 

4 runtime: python 

5 handlers: 

6 - url: /_ah/stats.* 

7 ■■■ 

8 - url: / (?P<a>.+?) /static/ (?P<b>.+) 

9 ... 

10 - url: /_ah/admin/.* 

12 - url: /_ah/queue/default 

13 ... 

14 - url: .* 

16 skip_files: 

17 ... 

app. example. yaml (when copied to app.yaml) is configured to deploy the 
web2py welcome application, but not the admin or example applications. You 
must replace web2py with the application id that you used when registering 
with Google App Engine. 

url: /(.+?)/static/(. + ) instructs GAE to serve your app static files directly, 
without calling web2py logic, for speed. 

url: .* instructs web2py to use the gaehandler.py for every other request. 

The skip_f iles : session is list of regular expressions for files that do not need 
to deployed on GAE. In particular the lines: 

1 (applications/ (admin | examples)/.*) | 
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( (admin I examples | welcome) \. (w2p| tar) ) | 

tell GAE not to deploy the default applications, except for the unpacked 
welcome scaffolding application. You can add more applications to be 
ignored here. 

Except for the application id and version, you probably do not need to edit 
app.yaml, though you may wish to exclude the welcome application. 

The file queue. yaml is used to configure GAE task queues. 

The file index. yaml is automatically generated when you run your application 
locally using the GAE appserver (the web server that comes with the Google 
SDK). It contains something like this: 

indexes: 

- kind: person 

properties: 

- name: name 
direction: desc 

In this example it tells GAE to create an index for table "person" that will be 
used to sort by "name" in reversed alphabetical order. You will not be able to 
search and sort records in your app without corresponding indexes. 

It is important to always run your apps locally with the appserver and try 
every functionality of your app, before deployment. This will be important 
for testing purposes, but also to automatically generate the "index. yaml" file. 
Occasionally you may want to edit this file and perform cleanup, such as 
removing duplicate entries. 

13.3.2 Running and deployment 

Linux 

Here we assume you have installed the GAE SDK. At the time of writing, 
GAE runs on Python 2.5.2. You can run your app from inside the "web2py" 
folder by using the appserver command: 

python2.5 dev_appserver.py . ./web2py 
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This will start the appserver and you can run your application at the URL: 


http://127. 0.0. 1:8080/ 


In order to upload your app on GAE, make sure you have edited the 
"app.yaml" file as explained before and set the proper application id, then 
run: 


python2.5 appcfg.py update ../web2py 


Mac, Windows 

On Mac and Windows, you can also use the Google App Engine Launcher. 
You can download the software from ref. [13]. 

Choose [File] [Add Existing Application], set the path to the path of the top- 
level weD2py folder, and press the [Run] button in the toolbar. After you have 
tested that it works locally, you can deploy it on GAE by simply clicking on 
the [Deploy] button on the toolbar (assuming you have an account). 
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On GAE, the web2py tickets /errors are also logged into the GAE 
administration console where logs can be accessed and searched online. 
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13.5.3 Configuring the handler 

The file gaehandler.py is responsible for serving files on GAE and it has a few 
options. Here are their default values: 

, L0G_STATS = False 
2 APPSTATS = True 
7 DEBUG = False 

L0G_STATS will log the time to serve pages in the GAE logs. 

APPSTATS will enable GAE appstats which provides profiling statistics. They 
will be made available at the URL: 

1 http://localhost:8080/_ah/stats 

DEBUG sets debug mode. It make no difference in practice unless checked 
explicitly in your code via gluon . settings .web2py_ runtime. 


13.5.4 Avoid the filesy stem 


On GAE you have no access to the filesystem. You cannot open any file for 
writing. 


DEPLOYMENT RECIPES 55I 


For this purpose, on GAE, web2py automatically stores all uploaded files in 
the datastore, whether or not "upload" Field(s) have a uploadfield attribute. 

You also should store sessions and tickets in the database and you have to be 
explicit: 

] if request .env.web2py_runtime_gae 
db = DAL( 'gae' ) 

3 session. connect (request, response, db) 

4 else: 

5 db = DAL( ' sqlite: //storage. sqlite' ) 

The above code checks whether you are running on GAE, connects to 
BigTable, and instructs web2py to store sessions and tickets in there. It 
connects to a sqlite database otherwise. This code is already in the scaffolding 
app in the file "db.py". 


13S -5 Memcache 

If you prefer, you can store sessions in memcache: 

1 from gluon. contrib.gae_memcache import MemcacheClient 

2 from gluon. contrib.memdb import MEMDB 

3 cache. memcache = MemcacheClient (request) 

4 cache. ram = cache. disk = cache. memcache 
5 

6 db = DAL( 'gae 1 ) 

7 session. connect (request, response, MEMDB (cache. memcache) ) 

Notice that on GAE cache. ram and cache. disk should not be used, so we 
make them point to cache. memcache. 


13.5.6 Datastore issues 

The absence of multi-entity transactions and typical functionalities of 
relational databases are what sets GAE apart from other hosting 
environment. This is the price to pay for high scalability. GAE is an excellent 
platform if these limitations are tolerable; if not, then a regular hosting 
platform with a relational database should be considered instead. 
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If a web2py application does not run on GAE, it is because of one of the 
limitations discussed above. Most issues can be resolved by removing JOINs 
from web2py queries and de-normalizing the database. 

Google App Engine supports some special field types, such as ListProperty 
and stringListProperty. You can use these types with web2py using the 
following old syntax: 

1 from gluon.dal import gae 

2 db. define. table ( 'product' , 

3 Field ( ' name ' ) , 

4 Field( ' tags' , type=gae.Stringl_istProperty( ) ) 

or the equivalent new syntax: 

1 db.define_table( 'product' , 

2 Field ( ' name ' ) , 

3 Field (' tags' , ' list : string' ) 

In both cases the "tags" field is a StringListProperty therefore its values must 
be lists of strings, compatibly with the GAE documentation. The second 
notation is to be preferred because web2py will treat the field in a smarter 
way in the context of forms and because it will work with relational databases 
too. 

Similarly, web2py supports list: integer and list: reference which map into 
a ListProperty(int). 

list types are discussed in more detail in Chapter 6. 


13.5.7 GAE and https 

If you application has id "myapp" your GAE domain is 

http : //myapp . appspot . com/ 

and it can also be accessed via HTTPS 

https : //myapp . appspot . com/ 

In this case it will use an "appspot.com" certificate provided by Google. 
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You can register a DNS entry and use any other domain name you own for 
your app but you will not be able to use HTTPS on it. At the time of writing, 
this is a GAE limitation. 


23.6 Jython 

web2py normally runs on CPython (the Python interpreter coded in C), but 
it can also run on Jython (the Python interpreter coded in Java). This allows 
web2py to run in a Java infrastructure. 

Even though web2py runs with Jython out of the box, there is some trickery 
involved in setting up Jython and in setting up zxJDBC (the Jython database 
adaptor). Here are the instructions: 

• Download the file "jython_installer-2.5.o.jar" (or 2.5.x) from Jython.org 

• Install it: 

Java -jar jython_installer-2.5.0. jar 

• Download and install "zxJDBC.jar" from [101] 

• Download and install the file "sqlitejdbc-vo56.jar" from [102] 

• Add zxJDBC and sqlitejdbc to the Java CLASSPATH 

• Start web2py with Jython 
/path/to/jython web2py.py 

At the time of writing we only support sqlite and postgres on Jython. 


Other recipes 


14.1 Upgrading 

In the "site" page of the administrative interface there is an "upgrade now" 
button. In case this is not feasible or does not work (for example because of 
a file locking issue), upgrading web2py manually is very easy. 

Simply unzip the latest version ofwebzpy over the old installation. 

This will upgrade all the libraries as well as the applications admin, 
examples, welcome. It will also create a new empty file "NEWINSTALL". 
Upon restarting, web2py will delete the empty file and package the welcome 
app into "welcome.w2p" that will be used as the new scaffolding app. 
web2py does not upgrade any file in other existing applications. 


24.2 How to distribute your applications as binaries 

It is possible to bundle your app with the web2py binary distribution and 
distribute them together. The license allows this as long you make clear in 
the license of your app that you are bundling with web2py and add a link to 
the web2py.com. 

Here we explain how to do it for Windows: 
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• Create your app as usual 

• Using admin, bytecode compile your app (one click) 

• Using admin, pack your app compiled (another click) 

• Create a folder "myapp" 

• Download a web2py windows binary distribution 

• Unzip it in folder "myapp" and start it (two clicks) 

• Upload using admin the previously packed and compiled app with the 
name "init" (one click) 

• Create a file "myapp/start.bat" that contains "cd web2py; web2py.exe" 

• Create a file "myapp/license" that contains a license for your app and 
make sure it states that it is being "distributed with an unmodified copy 
of web2py from web2py.c0m" 

• Zip the myapp folder into a file "myapp.zip" 

• Distribute and/or sell "myapp.zip" 

When users will unzip "myapp.zip" and click "run" they will see your app 
instead of the "welcome" app. There is no requirement on the user side, not 
even Python pre-installed. 

For Mac binaries the process is the same but there is no need for the "bat" 
file. 

24.3 Building a minimalist web2py 

Some times we need to deploy web2py in a server with very small memory 
footprint. In this case we want to strip down web2py to its bare minimum. 

An easy way to do it is the following: 

• On a production machine, install the full web2py from source 

• From inside the main web2py folder run 
python scripts/make_min_web2py .py /path/to/minweb2py 
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• Now copy under "/path/to/minweb2py/applications" the applications 
you want to deploy 

• Deploy "/path/to/minweb2py" to the small footprint server 

The script "make_min_web2pypy" builds a minimalist web2py distribution 
that does not include: 

• admin 

• examples 

• welcome 

• scripts 

• rarely used contrib modules 

It does include a "welcome" app consisting of a single file to allow testing 
deployment. Look into this script. At the top it contains a detailed list of 
what is included and what is ignored. You can easily modify it and tailor to 
your needs. 


24.4 Fetching an external URL 


Python includes the urllib library for fetching urls: 

import urllib 

page = urllib. urlopen( ' http://www. web2py. com' ). read( ) 


This is often fine, but the urllib module does not work on the Google App 
Engine. Google provides a different API for downloading URLs that works 
on GAE only. In order to make your code portable, web2py includes a fetch 
function that works on GAE as well as other Python installations: 

from google. tools import fetch 

page = fetchC http://www.web2py.com' ) 
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24.5 Pretty dates 

It is often useful to represent a datetime not as "2009-07-25 14:34:56" but as 
"one year ago". web2py provides a utility function for this: 

import datetime 

d = datetime. datetime(2009, 7,25,14, 34, 56) 
from gluon. tools import prettydate 
pretty_d = prettydate(d,T) 

The second argument (T) must be passed to allow internationalization for the 
output. 


24.6 Geocoding 

If you need to convert an address (for example: "243 S Wabash Ave, Chicago, 
IL, USA") into geographical coordinates (latitude and longitude), web2py 
provides a function to do so. 

from gluon. tools import geocode 

address = '243 S Wabash Ave, Chicago, IL, USA' 

(latitude, longitude) = geocode(address) 

The function geocode requires a network connection and it connects to the 
Google geocoding service for the geocoding. The function returns (0,0) in 
case of failure. Notice that the Google geocoding service caps the number of 
requests, so you should check their service agreement. The geocode function 
is built on top of the fetch function and thus it works on GAE. 


24.7 Pagination 

This recipe is a useful trick to minimize database access in case of pagination, 
e.g., when you need to display a list of rows from a database but you want 
to distribute the rows over multiple pages. 

Start by creating a primes application that stores the first 1000 prime numbers 
in a database. 
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Here is the model db. py: 

1 db = DAL( ' sqlite:// primes, db' ) 

2 db.define_table( 'prime' , Field ( ' value' , 'integer' ) ) 

3 def isprime(p) : 

4 for i in range(2,p) : 

5 if p%i==0: return False 

6 return True 

7 if len(db( ). select (db. prime. id) )==0: 
s p=2 

i, for i in range(1000): 
10 while not isprime(p): p+=l 

u db. prime. insert(value=p) 

p+=l 

Now create an action list-items in the "default. py" controller that reads like 
this: 



Notice that this code selects one more item than is needed, 20+1. The extra 
element tells the view whether there is a next page. 

Here is the "default/ list_items.html" view: 

1 {{extend 'layout.html'}} 
2 

3 {{for i,row in enumerate! rows) : }} 

4 {{if i==items_per_page: break}} 

5 {{=row.value}}<br /> 

6 {{pass}} 
7 

s {{if page:}} 

i, <a href="{{=URL(args=[page- I7,W>previous</a> 

.0 {{pass}} 

12 {{if len(rows)>items_per_page:}} 

13 <a href="{{=UfiZ-fargs=fpage+I7,W'>next</a> 
■4 {{pass}} 

In this way we have obtained pagination with one single select per action, 
and that one select only selects one row more then we need. 
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14.8 httpserver.log and the Log File Format 

The web2py web server logs all requests to a file called: 

httpserver.log 

in the root web2py directory. An alternative filename and location can be 
specified via web2py command-line options. 

New entries are appended to the end of the file each time a request is made. 
Each line looks like this: 

127.0.0.1, 2008-01-12 10:41:20, GET, /admin/default/site, HTTP/1.1, 200, 0.270000 

The format is: 

ip, timestamp, method, path, protocol, status, time_taken 

Where 

• ip is the IP address of the client who made the request 

• timestamp is the date and time of the request in ISO 8601 format, YYYY- 
MM-DDT HH:MM:SS 

• method is either GET or POST 

• path is the path requested by the client 

• protocol is the HTTP protocol used to send to the client, usually HTTP/ 1.1 

• status is the one of the HTTP status codes [95] 

• time_taken is the amount of time the server took to process the request, in 
seconds, not including upload /download time. 

In the appliances repository [34], you will find an appliance for log analysis. 

This logging is disabled by default when using mod_wsgi since it would be 
the same as the Apache log. 
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24.9 Populating database with dummy data 

For testing purposes it is convenient to be able to populate database tables 
with dummy data. web2py includes a Bayesian classifier already trained to 
generate dummy but readable text for this purpose. 

Here is the simplest way to use it: 

1 from gluon.contrib. populate import populate 

2 populate(db.mytable, 100) 

It will insert 100 dummy records into db.mytable. It will try to do 
intelligently by generating short text for string fields, longer text for text 
fields, integers, doubles, dates, datetimes, times, booleans, etc. for the 
corresponding fields. It will try to respect requirements imposed by 
validators. For fields containing the word "name" it will try to generate 
dummy names. For reference fields it will generate valid references. 

If you have two tables (A and B) where B references A, make sure to populate 
A first and B second. 

Because population is done in a transaction, do not attempt to populate 

too many records at once, particularly if references are involved. Instead, 

populate 100 at a time, commit, loop. 

] for i in range(10) : 

z populate(db.mytable, 100) 

3 db. commit ( ) 

You can use the Bayesian classifier to learn some text and generate dummy 
text that sounds similar but should not make sense: 

1 from gluon.contrib. populate import Learner, IUP 

2 ell=l_earner( ) 

3 ell. learnt 'some very long input text ...') 

4 print ell .generate(1000,prefix=None) 


24.20 Accepting credit card payments 

There are multiple ways to accept credit card payments online. web2py 
provides specific APIs for some of the most popular and practical ones: 
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• Google Wallet [96] 

• PayPal paypalcite 

• Stripe.com [98] 

• Authorize.net [99] 

• DowCommerece [100] 

The first two mechanisms above delegate the process of authenticating the 
payee to an external service. While this is the best solution for security (your 
app does not handle any credit card information at all) it makes the process 
cumbersome (the user must login twice; for example, once with your app, 
and once with Google) and does not allow your app to handle recurrent 
payments in an automated way. 

There are times when you need more control and you want to generate 
yourself the entry form for the credit card info and than programmatically 
ask the processor to transfer money from the credit card to your account. 

For this reason web2py provide integration out of the box with Stripe, 
Authorize.net (the module was developed by John Conde and slightly 
modified) and DowCommerce. Stripe is the simplest to use and also the 
cheapest for low volume of transactions (they charge no fix cost but charge 
about 3% per transaction). Authorize.net is better for high volumes (has a 
fixed yearly costs plus a lower cost per transaction). 

Mind that in the case of Stripe and Authorize.net your program will be 
accepting credit cards information. You do not have to store this information 
and we advice you not to because of the legal requirements involved (check 
with Visa or Mastercard) but there are times when you may want to store the 
information for recurrent payments or to reproduce the Amazon one-click 
pay button. 

14.10.1 Google Wallet 

The simplest way to use Google Wallet (Level 1) consists of embedding a 
button on your page that, when clicked, redirects your visitor to a payment 


OTHER RECIPES 563 

page provided by Google. 

First of all you need to register a Google Merchant Account at the url: 

https://checkout.google.com/sell 

You will need to provide Google with your bank information. Google will 
assign you a merchant-id and a merchant-key (do not confuse them, keep them 
secret). 


Then 

you 

simply 

need to create the following 

code in 

your 

view: 

{{from 

gluon . cont rib . google_ wallet import 

button}} 




{{=button(r 

nerchant_ 

id=" 123456789012345" , 







products 

=[dict (name=" shoes" , 
quantity=l, 
price=23.5, 
currency=' USD' , 








description^" running shoes black' 

')])}} 



When a visitor clicks on the button, the visitor will be redirected to the 
Google page where he/she can pay for the items. Here products is a list 
of products and each product is a dictionary of parameters that you want 
to pass describing your items (name, quantity, price, currency, description, 
and other optional ones which you can find described in the Google Wallet 
documentation) . 

If you choose to use this mechanism you may want to generate the values 
passed to the button programmatically based on your inventory and the 
visitor shopping chart. 

All the tax and shipping information will be handled on the Google side. 
Same for accounting information. By default your application is not notified 
that the transaction has been completed therefore you will have to visit your 
Google Merchant site to see which products have been purchased and paid 
for, and which products you need to ship to your buyers there. Google will 
also send you an email with the information. 

If you want a tighter integration you have to use the Level 2 notification API. 
In that case you can pass more information to Google and Google will call 
your API to notify about purchases. This allows you to keep accounting 
information within your application but it requires you expose web services 
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that can talk to Google Wallet. 

This is a considerable more difficult problem but such API has already been 
implemented and it is available as plugin from 

http://web2py . com/plugins/s t at ic/web2py. plugin . google_ checkout .w2p 
You can find the documentation of the plugin in the plugin itself. 


14.10.2 Paypal 

Paypal integration is not described here but you can find more information 
about it at this resource: 

http://www.web2pyslices.eom/main/slices/take_slice/9 


14.10.3 Stripe.com 

This is probably one of the easiest way and flexible ways to accept credit card 
payments. 

You need to register with Stripe.com and that is a very easy process, in 
fact Stripe will assign you an API key to try even before you create any 
credentials. 

Once you have the API key you can accept credit cards with the following 
code: 

1 from gluon. contrib. stripe import Stripe 

2 stripe = Stripe(api_key) 

3 d = stripe. charge(amount=100 : 

4 currency='u_d ' , 

card_number=' 4242424242424242' , 

6 card_exp_month='5' , 

7 ca rd_exp_yea r= ' 2012 ' , 
s ca rd_cvc_check=' 123 ' , 

9 description^' the usual black shoes') 

n, if d.get( 'paid' , False) : 

u # payment accepted 

iz elif: 

i 3 # error is in d.getf error' ,' unknown' ) 
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The response, d, is a dictionary which you can explore yourself. The card 
number used in the example is a sandbox and it will always succeed. Each 
transaction is associated to a transaction id stored in d [ ' id ' ] . 

Stripe also allows you verify a transaction at a later time: 

] d = Stripe(key) .check(d[ 'id' ] ) 

and refund a transaction: 

] r = Stripe(key) . refund(d[ 'id' ] ) 
z if r. get (' refunded ' .False) : 

3 # refund was successful 

4 elif: 

5 # error is in d.getf error' , 'unkown' ) 

Stripe makes very easy to keep the accounting within your application. 

All the communications between your app and Stripe go over RESTful web 
services. Stripe actually exposes even more services and provides a larger set 
of Python API. You can read more on their web site. 


14.10.4 Authorize.Net 

Another simple way to accept credit cards is to use Authorize.Net. As 
usual you need to register and you will obtain a login and a transaction 
key (transkey. Once you have them it works very much like Stripe does: 



If you have a valid Authorize.Net account you should replace the sandbox 
login and transkey with those of your account, set testmode=False to run on 
the real platform instead of the sandbox, and use credit card information 
provided by the visitor. 
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If process returns True, the money has been transferred from the visitor credit 
card account to your Authorize. Net account, invoice is just a string that you 
can set and will be store by Authorize. Net with this transaction so that you 
can reconcile the data with the information in your application. 

Here is a more complex example of workflow where more variables are 
exposed: 


from gluon. contrib.AuthorizeNet import AIM 
payment = AIM ( login= ' cnpdev4289' , 

transkey=' SR2P8g4jdEn7vFLQ' , 

testmod=True) 
payment .setTran sac t ion (credit card, expiration, 


total, cvv, tax, invoice) 


payment .setParameter 
payment .setParameter 
payment .setParameter 
payment .setParameter 
payment .setParameter 
payment .setParameter 
payment .setParameter 
payment . set Pa ramete r 
payment . set Pa ramete r 
payment .setParameter 
payment .setParameter 
payment . set Pa ramete r 
payment . set Pa ramete r 
payment .setParameter 
payment .setParameter 


# three minutes duplicate windows 

# customer ID 


x-duplicate^window' , 180) 

X-Cust-id' , '1324') 

x-first-name' , 'Agent') 

x-last-name' , 'Smith') 

x-company' , 'Test Company') 

x_address' , '1234 Main Street') 

X-City' , 'Townsville' ) 

x_state' , 'NJ') 

x^zip ' , ' 12345 ' ) 

x_country' , 'US' ) 

x-phone' , '880-555-1234') 

x_description' , Test Transaction' ) 

x-customer^ip' , socket . gethostbyname( socket .gethostname( ) ) ) 

x_email' , 'you@example.com') 

x-email-customer' , False) 


payment .process! ) 

if payment . isApproved( ) : 

print 'Response Code: ' , payment . response. ResponseCode 
'Response Text: ', payment . response. ResponseText 
'Response: ', payment .getResultResponseFull( ) 
'Transaction ID: ' , payment . response. TransactionID 
'CVV Result: ', payment . response. CVVResponse 
'Approval Code: ', payment . response. AuthCode 
'AVS Result: ', payment . response. AVSResponse 
nt .isDeclined( ) : 
'Your credit card was declined by your bank' 

33 elif payment. isError( ) : 

34 print 'It did not work' 

35 print 'approved' .payment .isApp roved ( ) 

3 6 print 'declined' .payment .isDeclined( ) 

37 print 'error' .payment .isError( ) 


5 

print 

6 

print 

7 

print 

8 

print 

9 

print 



print 

1 

elif paym 

2 

print 
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Notice the code above uses a dummy test account. You need to register with 
Authorize. Net (it is not a free service) and provide your own login, transkey 
testmode=True or False to the AIM constructor. 


24.22 Dropbox API 

Dropbox is a very popular storage service. It not only stores your files but 
it keeps the cloud storage in sync with all your machines. It allows you 
to create groups and give read /write permissions to the various folders to 
individual users or groups. It also keeps version history of all your files. It 
includes a folder called "Public" and each file you put in there will have its 
own public URL. Dropbox is a great way to collaborate. 

You can access dropbox easily by registering at 
https : //www . d ropbox . com/develope rs 

you will get an APP_KEY and an APP_SECRET. Once you have them you can use 
Dropbox to authenticate your users. 

Create a file called "yourapp/private/dropbox.key" and in it write 

<APP_KEY> : <APP_SECERT> : app_f olde r 


where <APP_KEY> and APP_SECRET are your key and secret. 

Then in "models/db.py" do: 

1 from gluon. contrib.logiri_methods.dropbox_account import use_dropbox 
z use_ j anrain(auth, f ilename= ' private/ dropbox. key ' ) 
3 mydropbox = auth. settings .login_form 

This will allow users to login into your app using their dropbox credentials, 
and your program will be able to upload files into their dropbox account: 

1 stream = open( ' localfile. txt ' , ' rb' ) 

2 myd ropbox. put ( ' destfile. txt' , stream) 

download files: 

1 stream = myd ropbox. get ( 'destfile. txt ' ) 

2 open( ' localfile. txt' ,'wb') .write) read) 
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and get directory listings: 

contents = mydropbox.dir(path = '/')[' contents' ] 


14.12 Twitter API 

Here are some quick examples on how to post /get tweets. No third-party 
libraries are required, since Twitter uses simple RESTful APIs. 

Here is an example of how to post a tweet: 

def post_ tweet (username, password, message) : 
import urllib, urllib2, base64 
import gluon.contrib. simplejson as sj 
args= urllib. urlencode( [ ( ' status ' .message) ] ) 
headers={} 
headers [ 'Authorization' ] = 'Basic '+base64.b64encode( 

username+' : '+password) 
req = urllib2.Request( 

'http://twitter.com/statuses/update.json' , 

args, headers) 
return sj . loads (urllib2.urlopen( req) . read( ) ) 

Here is an example of how to receive tweets: 

def get_ tweets ( ) : 
user= 'web2py ' 
import urllib 

import gluon.contrib. simplejson as sj 

page = urllib. urlopen( ' http: / /twitter. com/%s?format=json' % user).read() 
tweets=XML(sj .loads (page) [ '#timeline' ] ) 
return diet (tweets=tweets) 

For more complex operations, refer to the Twitter API documentation. 


24.23 Streaming virtual files 

It is common for malicious attackers to scan web sites for vulnerabilities. 
They use security scanners like Nessus to explore the target web sites for 
scripts that are known to have vulnerabilities. An analysis of web server 
logs from a scanned machine or directly in the Nessus database reveals that 
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most of the known vulnerabilities are in PHP scripts and ASP scripts. Since 
we are running web2py, we do not have those vulnerabilities, but we will 
still be scanned for them. This is annoying, so we like to respond to those 
vulnerability scans and make the attacker understand their time is being 
wasted. 

One possibility is to redirect all requests for.php,.asp, and anything 
suspicious to a dummy action that will respond to the attack by keeping 
the attacker busy for a large amount of time. Eventually the attacker will 
give up and will not scan us again. 

This recipe requires two parts. 

A dedicated application called jammer with a "default.py" controller as 
follows: 

class Jammer( ) : 

def read(self ,n) : return 'x'*n 
def jam(): return response. stream(Jammer( ) ,40000) 

When this action is called, it responds with an infinite data stream full of 
"x"-es. 40000 characters at a time. 

The second ingredient is a "route. py" file that redirects any request ending 

in.php,.asp, etc. (both upper case and lower case) to this controller. 

route_in=( 
('.*!. (php\PHP\asp\ASP\jsp\JSP) ' , ' jammer/ default/ jam' ) , 


3 


The first time you are attacked you may incur a small overhead, but our 
experience is that the same attacker will not try twice. 
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